Nov 03 18:41:36.473279 ip-10-0-139-179 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 18:41:36.473288 ip-10-0-139-179 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 18:41:36.473294 ip-10-0-139-179 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 18:41:36.473612 ip-10-0-139-179 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 18:41:46.703405 ip-10-0-139-179 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 18:41:46.703422 ip-10-0-139-179 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8baf9dd70a0c4fc8920b0fdb6e336fae -- Nov 03 18:43:42.800667 ip-10-0-139-179 systemd[1]: Starting Kubernetes Kubelet... Nov 03 18:43:43.249087 ip-10-0-139-179 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:43:43.249087 ip-10-0-139-179 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 18:43:43.249087 ip-10-0-139-179 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:43:43.249087 ip-10-0-139-179 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:43:43.253424 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253319 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:43:43.253424 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253419 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:43:43.253424 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253424 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:43:43.253424 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253429 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:43:43.253424 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253432 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253437 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253441 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253445 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253448 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253452 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253455 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253461 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253464 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253468 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253472 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253476 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253480 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253484 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253488 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253492 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253496 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253500 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253504 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:43:43.253726 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253508 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253511 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253525 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253530 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253534 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253538 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253542 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253546 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253550 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253554 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253559 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253563 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253568 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253572 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253577 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253582 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253587 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253591 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253595 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253602 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:43:43.254510 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253609 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253614 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253618 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253623 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253628 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253633 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253637 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253647 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253652 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253656 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253698 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253705 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253710 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253714 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253719 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253724 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253728 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253733 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253737 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253742 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:43:43.255312 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253746 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253750 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253754 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253759 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253763 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253768 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253772 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253777 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253781 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253786 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253790 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253794 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253799 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253804 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253808 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253813 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253818 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253822 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253826 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253830 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:43:43.256192 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253834 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253838 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253843 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253847 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253851 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253855 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253859 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253867 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253876 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253882 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253887 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253891 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253905 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253911 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253916 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253920 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253925 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253930 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253934 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:43:43.257016 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253938 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253942 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253947 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.253951 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255206 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255218 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255225 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255230 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255235 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255240 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255244 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255249 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255253 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255257 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255261 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255266 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255270 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255274 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255278 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255283 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:43:43.257689 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255287 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255291 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255296 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255302 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255306 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255310 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255314 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255318 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255322 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255343 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255347 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255352 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255356 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255360 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255364 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255368 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255372 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255376 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255381 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255385 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:43:43.258169 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255389 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255393 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255397 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255401 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255405 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255410 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255415 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255421 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255425 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255429 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255434 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255438 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255442 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255446 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255450 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255455 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255459 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255464 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255469 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:43:43.258662 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255474 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255481 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255487 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255492 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255497 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255501 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255506 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255510 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255514 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255518 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255523 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255527 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255531 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255535 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255539 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255543 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255548 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255552 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255556 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255560 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:43:43.259143 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255565 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255569 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255573 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255577 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255581 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255585 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255592 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255597 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255601 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255606 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255610 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255614 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255619 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255623 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255627 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255632 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255636 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255640 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255644 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:43:43.259631 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255649 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255653 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255657 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255661 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255665 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255670 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255674 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255678 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255684 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255688 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255692 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.255696 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257128 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257140 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257145 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257150 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257156 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257166 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257171 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:43:43.260084 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257179 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257186 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257192 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257198 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257204 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257209 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257213 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257218 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257222 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257227 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257231 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257241 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257246 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257250 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257255 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257259 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257264 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257268 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257273 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:43:43.260552 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257277 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257282 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257286 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257290 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257300 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257305 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257309 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257314 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257318 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257322 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257342 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257346 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257351 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257365 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257370 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257376 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257383 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257389 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257394 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257399 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:43:43.260994 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257403 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257408 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257412 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257417 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257421 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257430 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257435 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257439 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257443 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257448 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257452 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257457 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257461 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257465 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257470 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257609 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257621 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257630 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257635 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:43:43.261476 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257646 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257651 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257655 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257659 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257663 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257668 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257672 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257676 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257680 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257685 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257688 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257697 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257701 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257706 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257710 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257714 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257718 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257722 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257726 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257730 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:43:43.261930 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257734 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257738 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257742 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257747 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257751 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257755 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257762 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257772 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257783 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257787 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257792 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257798 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257802 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257807 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257811 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257815 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257819 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257824 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257827 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257830 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:43:43.262429 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.257834 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258634 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258646 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258653 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258658 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258662 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258666 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258671 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258675 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258678 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258681 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258685 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258688 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258691 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258694 2579 flags.go:64] FLAG: --cgroup-root="" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258697 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258701 2579 flags.go:64] FLAG: --client-ca-file="" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258703 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258707 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258712 2579 flags.go:64] FLAG: --cluster-domain="" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258717 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258720 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258724 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258728 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 18:43:43.262876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258732 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258735 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258738 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258741 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258744 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258747 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258750 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258753 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258756 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258761 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258764 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258767 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258769 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258772 2579 flags.go:64] FLAG: --enable-server="true" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258775 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258781 2579 flags.go:64] FLAG: --event-burst="100" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258784 2579 flags.go:64] FLAG: --event-qps="50" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258787 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258790 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258793 2579 flags.go:64] FLAG: --eviction-hard="" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258798 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258800 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258804 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258807 2579 flags.go:64] FLAG: --eviction-soft="" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258810 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 18:43:43.263438 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258813 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258816 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258819 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258823 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258826 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258830 2579 flags.go:64] FLAG: --feature-gates="" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258834 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258837 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258841 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258844 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258847 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258850 2579 flags.go:64] FLAG: --help="false" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258853 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258856 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258859 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258862 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258865 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258869 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258871 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258874 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258877 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258880 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258884 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 18:43:43.264075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258887 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258890 2579 flags.go:64] FLAG: --kube-reserved="" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258893 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258896 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258899 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258902 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258905 2579 flags.go:64] FLAG: --lock-file="" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258908 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258911 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258914 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258920 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258923 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258927 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258930 2579 flags.go:64] FLAG: --logging-format="text" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258933 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258937 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258939 2579 flags.go:64] FLAG: --manifest-url="" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258943 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258947 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258951 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258955 2579 flags.go:64] FLAG: --max-pods="110" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258957 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258961 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258963 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258966 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 18:43:43.264630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258969 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258972 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258975 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258983 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258986 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258989 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258992 2579 flags.go:64] FLAG: --pod-cidr="" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258995 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.258997 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259000 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259003 2579 flags.go:64] FLAG: --port="10250" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259006 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259010 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0fa26096d4518dd05" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259014 2579 flags.go:64] FLAG: --qos-reserved="" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259017 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259020 2579 flags.go:64] FLAG: --register-node="true" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259023 2579 flags.go:64] FLAG: --register-with-taints="" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259026 2579 flags.go:64] FLAG: --registry-burst="10" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259029 2579 flags.go:64] FLAG: --registry-qps="5" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259033 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259036 2579 flags.go:64] FLAG: --reserved-memory="" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259040 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259044 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259047 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259049 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259052 2579 flags.go:64] FLAG: --runonce="false" Nov 03 18:43:43.265227 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259055 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259058 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259061 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259064 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259067 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259070 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259073 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259076 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259079 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259082 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259085 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259088 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259091 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259094 2579 flags.go:64] FLAG: --system-cgroups="" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259096 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259103 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259106 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259108 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259114 2579 flags.go:64] FLAG: --tls-min-version="" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259117 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259120 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259123 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259126 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259129 2579 flags.go:64] FLAG: --v="2" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259133 2579 flags.go:64] FLAG: --version="false" Nov 03 18:43:43.265882 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259139 2579 flags.go:64] FLAG: --vmodule="" Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259143 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259147 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259276 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259281 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259284 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259287 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259291 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259294 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259297 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259300 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259303 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259305 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259308 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259311 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259313 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259316 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259319 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259321 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259336 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:43:43.266491 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259339 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259341 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259344 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259348 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259352 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259358 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259361 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259364 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259367 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259370 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259373 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259376 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259380 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259383 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259385 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259388 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259391 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259393 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259396 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:43:43.266963 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259399 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259402 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259404 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259407 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259410 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259412 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259415 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259417 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259420 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259423 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259425 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259428 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259430 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259433 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259436 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259438 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259441 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259443 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259447 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259450 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:43:43.267453 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259453 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259455 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259458 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259461 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259463 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259467 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259470 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259472 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259475 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259478 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259480 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259483 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259486 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259488 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259491 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259493 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259496 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259499 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259501 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259504 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:43:43.267954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259507 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259509 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259512 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259514 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259517 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259520 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259522 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259525 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259527 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259530 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259534 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259536 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259539 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259542 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259544 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259547 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259550 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259554 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259557 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259559 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:43:43.268431 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259562 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259564 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259567 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259570 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259572 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259575 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259577 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259580 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259582 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259585 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.259611 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259647 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259652 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259655 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259657 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:43:43.268891 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259660 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259663 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259665 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259668 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259671 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259673 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259676 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259679 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259682 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259685 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259688 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259691 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259693 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259697 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259701 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259703 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259706 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259709 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259711 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:43:43.269245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259714 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259716 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259719 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259722 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259724 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259727 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259729 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259732 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259734 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259737 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259739 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259742 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259745 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259747 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259750 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259752 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259755 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259757 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259760 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259763 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:43:43.269711 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259765 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259769 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259771 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259774 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259777 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259779 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259782 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259786 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259789 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259792 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259795 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259797 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259800 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259802 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259805 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259808 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259810 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259818 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259821 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259823 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:43:43.270183 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259826 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259828 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259831 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259834 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259836 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259839 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259841 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259844 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259847 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259849 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259852 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259854 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259857 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259862 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259866 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259869 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259872 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259874 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259877 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:43:43.270665 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259881 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259884 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259887 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259889 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259892 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259894 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259897 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259900 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259902 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259905 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259908 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259910 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259913 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259915 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259920 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259924 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259927 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259930 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259932 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:43:43.271114 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259935 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259938 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259940 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259943 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.259946 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.266976 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.266992 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.267004 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.267009 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267063 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267068 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267071 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267075 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267079 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267083 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267086 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267089 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267092 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267094 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:43:43.271722 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267097 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267100 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267102 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267105 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267108 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267111 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267113 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267116 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267119 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267121 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267124 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267127 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267129 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267132 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267135 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267138 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267140 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267143 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267146 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267150 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:43:43.272166 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267153 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267155 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267158 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267161 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267163 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267166 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267169 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267171 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267174 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267177 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267179 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267182 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267185 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267187 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267190 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267192 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267195 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267197 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267200 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267202 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:43:43.272648 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267205 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267208 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267210 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267213 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267215 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267218 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267221 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267224 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267227 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267230 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267232 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267235 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267238 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267241 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267244 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267246 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267249 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267253 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267256 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:43:43.273167 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267259 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267262 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267265 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267267 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267270 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267273 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267276 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267278 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267280 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267283 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267286 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267289 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267291 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267294 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267297 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267299 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267302 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267304 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267307 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267310 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:43:43.273667 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267313 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267315 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267318 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267321 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267337 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267342 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267346 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267349 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267351 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267354 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267357 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267359 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267362 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267364 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267367 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267369 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:43:43.274208 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267372 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.267399 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267537 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267542 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267545 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267548 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267552 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267555 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267558 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267560 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267563 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267566 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267568 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267571 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267574 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267576 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:43:43.274602 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267579 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267581 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267584 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267587 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267589 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267592 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267595 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267598 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267600 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267603 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267605 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267608 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267610 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267613 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267616 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267618 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267621 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267623 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267626 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267628 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:43:43.275010 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267631 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267633 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267636 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267639 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267641 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267644 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267646 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267649 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267651 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267654 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267657 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267659 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267662 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267664 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267667 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267669 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267672 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267675 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267677 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267680 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:43:43.275566 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267682 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267685 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267688 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267690 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267693 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267695 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267698 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267700 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267703 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267705 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267709 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267713 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267716 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267719 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267722 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267724 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267727 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267729 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267732 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:43:43.276354 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267734 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267737 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267739 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267742 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267745 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267747 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267750 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267752 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267755 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267758 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267760 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267763 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267766 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267770 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267774 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267777 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267779 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267782 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267784 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:43:43.276954 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267787 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267789 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267792 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267795 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267797 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267800 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267802 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267805 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267807 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267810 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267812 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267815 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267817 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267820 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.267842 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267873 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:43:43.277420 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267877 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267880 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267883 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267886 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267889 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267893 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267896 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267898 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267902 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267904 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267907 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267911 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267915 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267918 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267921 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267924 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267926 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267929 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267931 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:43:43.277855 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267934 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267936 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267939 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267942 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267944 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267947 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267949 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267952 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267954 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267957 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267960 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267963 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267966 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267968 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267971 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267974 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267976 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267979 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267981 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:43:43.278298 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267984 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267986 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267989 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267991 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267994 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267996 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.267999 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268009 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268012 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268014 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268017 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268020 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268023 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268025 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268027 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268030 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268033 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268035 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268038 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268043 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:43:43.278981 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268046 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268050 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268053 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268055 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268058 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268060 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268063 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268066 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268069 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268071 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268074 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268077 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268079 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268082 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268085 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268087 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268090 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268092 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268095 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268097 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:43:43.279508 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268108 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268110 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268113 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268116 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268118 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268121 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268123 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268126 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268128 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268131 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268134 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268137 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268140 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268144 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268147 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268149 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268152 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268155 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268159 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:43:43.280028 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268162 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268165 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268168 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268171 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268173 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268176 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268178 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:43.268181 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.268235 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.272482 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.273528 2579 server.go:1014] "Starting client certificate rotation" Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.273638 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.273689 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.276597 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:43:43.280494 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.277660 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:43:43.287525 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.287507 2579 log.go:25] "Validated CRI v1 runtime API" Nov 03 18:43:43.291276 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.291262 2579 log.go:25] "Validated CRI v1 image API" Nov 03 18:43:43.292114 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.292088 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 18:43:43.298172 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.298155 2579 fs.go:135] Filesystem UUIDs: map[72112ea2-b8bf-43ab-95b4-895866284f31:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e70b221b-ded6-42c1-8a22-aa22a3c39021:/dev/nvme0n1p4] Nov 03 18:43:43.298243 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.298173 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 18:43:43.304010 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.303903 2579 manager.go:217] Machine: {Timestamp:2026-11-03 18:43:43.303435947 +0000 UTC m=+0.374466290 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099168 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec227d25bf2188b50acebe17e5b51ab7 SystemUUID:ec227d25-bf21-88b5-0ace-be17e5b51ab7 BootID:8baf9dd7-0a0c-4fc8-920b-0fdb6e336fae Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:fb:6d:3f:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:fb:6d:3f:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:db:0a:ca:37:a2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 18:43:43.304010 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.303999 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 18:43:43.304146 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.304092 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 18:43:43.307549 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.307524 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 18:43:43.307671 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.307550 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-179.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 18:43:43.307712 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.307673 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 18:43:43.307712 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.307681 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 18:43:43.307712 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.307703 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:43:43.307787 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.307721 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:43:43.308983 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.308973 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 18:43:43.313509 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.313497 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 18:43:43.313618 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.313608 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 18:43:43.314422 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.314412 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 03 18:43:43.314469 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.314425 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 18:43:43.314469 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.314438 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 18:43:43.314469 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.314449 2579 kubelet.go:405] "Adding apiserver pod source" Nov 03 18:43:43.314469 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.314463 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 18:43:43.314776 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.314760 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:43:43.317277 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.317257 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 18:43:43.324590 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.324577 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:43:43.325460 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.325448 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 18:43:43.325498 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.325478 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 18:43:43.325630 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.325620 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 18:43:43.325673 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.325636 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 18:43:43.325673 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.325649 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 18:43:43.325673 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.325655 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 18:43:43.325673 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.325662 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 18:43:43.325673 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.325672 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 18:43:43.326425 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.326416 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 18:43:43.326455 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.326426 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 18:43:43.326455 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.326433 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 18:43:43.326455 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.326439 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 18:43:43.326455 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.326447 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 18:43:43.326455 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.326456 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 18:43:43.327371 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.327361 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 18:43:43.327405 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.327374 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 18:43:43.331121 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.331110 2579 server.go:1263] "Started kubelet" Nov 03 18:43:43.331341 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.331237 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 18:43:43.331408 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.331358 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 18:43:43.331457 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.331352 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 18:43:43.331873 ip-10-0-139-179 systemd[1]: Started Kubernetes Kubelet. Nov 03 18:43:43.333136 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.333119 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 18:43:43.335119 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.335106 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 03 18:43:43.338986 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.338968 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 18:43:43.341105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.341085 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 18:43:43.341105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.341104 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 18:43:43.341240 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.341187 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-179.us-west-2.compute.internal\" not found" Nov 03 18:43:43.341921 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.341901 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 18:43:43.342012 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.341977 2579 factory.go:55] Registering systemd factory Nov 03 18:43:43.342012 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.341991 2579 factory.go:223] Registration of the systemd container factory successfully Nov 03 18:43:43.342114 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.342037 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 18:43:43.342114 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.342063 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 18:43:43.342197 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.342163 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 03 18:43:43.342197 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.342173 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 03 18:43:43.342286 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.342271 2579 factory.go:153] Registering CRI-O factory Nov 03 18:43:43.342349 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.342288 2579 factory.go:223] Registration of the crio container factory successfully Nov 03 18:43:43.342388 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.342360 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 18:43:43.342388 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.342385 2579 factory.go:103] Registering Raw factory Nov 03 18:43:43.342471 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.342402 2579 manager.go:1199] Started watching for new ooms in manager Nov 03 18:43:43.344609 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.344587 2579 manager.go:319] Starting recovery of all containers Nov 03 18:43:43.356177 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.356159 2579 manager.go:324] Recovery completed Nov 03 18:43:43.360641 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.360629 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:43:43.362276 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.362255 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:43:43.362346 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.362288 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:43:43.362346 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.362299 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:43:43.363663 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.363644 2579 cpu_manager.go:226] "Starting" policy="none" Nov 03 18:43:43.363663 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.363657 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:43:43.363808 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.363659 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 18:43:43.363808 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.363688 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 18:43:43.364487 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.364471 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:43:43.364565 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.364501 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:43:43.364565 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.364516 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:43:43.366206 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.366189 2579 policy_none.go:50] "Start" Nov 03 18:43:43.366265 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.366209 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 18:43:43.366265 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.366228 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 18:43:43.368275 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.368263 2579 policy_none.go:44] "Start" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.407884 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.410091 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.410580 2579 manager.go:341] "Starting Device Plugin manager" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.410613 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.410625 2579 server.go:87] "Starting device plugin registration server" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.410897 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.410910 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.411263 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.411355 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.411363 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.411557 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.411574 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.411593 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.411642 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.412201 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 18:43:43.418579 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.412247 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-179.us-west-2.compute.internal\" not found" Nov 03 18:43:43.482740 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.481560 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-179.us-west-2.compute.internal.18e49d39d61ad0d0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-179.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-179.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:43:43.331078352 +0000 UTC m=+0.402108695,LastTimestamp:2026-11-03 18:43:43.331078352 +0000 UTC m=+0.402108695,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-179.us-west-2.compute.internal,}" Nov 03 18:43:43.512064 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.512012 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:43:43.512176 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.512010 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal"] Nov 03 18:43:43.512176 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.512148 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:43:43.513868 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.513849 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:43:43.513966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.513886 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:43:43.513966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.513901 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:43:43.513966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.513856 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:43:43.513966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.513952 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:43:43.513966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.513963 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:43:43.514195 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.513930 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.516163 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.516149 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:43:43.516276 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.516261 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.516348 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.516299 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:43:43.516856 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.516841 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:43:43.516856 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.516850 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:43:43.516981 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.516868 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:43:43.516981 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.516873 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:43:43.516981 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.516885 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:43:43.516981 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.516910 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:43:43.518923 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.518909 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.518982 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.518933 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:43:43.520200 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.520187 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:43:43.520269 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.520214 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:43:43.520269 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.520229 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:43:43.542799 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.542585 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d9f7f968eecc0f1a776909177a87d50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"9d9f7f968eecc0f1a776909177a87d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.542886 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.542821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d9f7f968eecc0f1a776909177a87d50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"9d9f7f968eecc0f1a776909177a87d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.542886 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.542856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1997d138e76444593b76299593de1e7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"b1997d138e76444593b76299593de1e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.544780 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.544764 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-179.us-west-2.compute.internal\" not found" node="ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.546667 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.546651 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-179.us-west-2.compute.internal\" not found" node="ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.558600 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.558583 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 18:43:43.558658 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.558619 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-79sbc" Nov 03 18:43:43.570545 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.570524 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-179.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 18:43:43.570812 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.570786 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-179.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 18:43:43.634982 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.634960 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:43:43.635214 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.635198 2579 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-179.us-west-2.compute.internal\" not found" node="ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.635571 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.635556 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:43:43.636385 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.636373 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:43:43.643885 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.643872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d9f7f968eecc0f1a776909177a87d50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"9d9f7f968eecc0f1a776909177a87d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.643938 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.643892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1997d138e76444593b76299593de1e7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"b1997d138e76444593b76299593de1e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.643938 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.643908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d9f7f968eecc0f1a776909177a87d50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"9d9f7f968eecc0f1a776909177a87d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.644032 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.643952 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d9f7f968eecc0f1a776909177a87d50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"9d9f7f968eecc0f1a776909177a87d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.644032 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.643972 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d9f7f968eecc0f1a776909177a87d50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"9d9f7f968eecc0f1a776909177a87d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.644032 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.643992 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1997d138e76444593b76299593de1e7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"b1997d138e76444593b76299593de1e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.728504 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.728481 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:43:43.836042 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.835982 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:43:43.836941 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.836923 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:43:43.837012 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.836954 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:43:43.837012 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.836963 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:43:43.837012 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.836990 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.850390 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.850363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.852526 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.852509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.868450 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.868432 2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-179.us-west-2.compute.internal\" not found" node="ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.891523 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.891501 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:43.891523 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:43.891522 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-179.us-west-2.compute.internal\": node \"ip-10-0-139-179.us-west-2.compute.internal\" not found" Nov 03 18:43:43.941603 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:43.941582 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:44.008043 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.008022 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:43:44.008135 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.008124 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:43:44.112299 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.112229 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:43:44.273403 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.273372 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 18:43:44.273958 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.273501 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 18:43:44.273958 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.273531 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 18:43:44.273958 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.273536 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 18:43:44.273958 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:44.273591 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3904a6b916d941eabd9d7e63c223c68-8c1f0edaac096962.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-179.us-west-2.compute.internal.18e49d39d7f6db99\": read tcp 10.0.139.179:52082->3.131.114.228:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-179.us-west-2.compute.internal.18e49d39d7f6db99 default 4766 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-179.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-179.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-179.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:43:43 +0000 UTC,LastTimestamp:2026-11-03 18:43:43.513939948 +0000 UTC m=+0.584970291,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-179.us-west-2.compute.internal,}" Nov 03 18:43:44.314626 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.314602 2579 apiserver.go:52] "Watching apiserver" Nov 03 18:43:44.341740 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.341713 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 18:43:44.564800 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.564759 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 18:38:43 +0000 UTC" deadline="2028-05-28 21:32:36.388900765 +0000 UTC" Nov 03 18:43:44.564800 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.564790 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13730h48m51.824113339s" Nov 03 18:43:44.601017 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.600991 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 18:43:44.602138 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.601652 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-dwp5g","openshift-dns/node-resolver-dlbxv","openshift-multus/multus-rwwj5","kube-system/global-pull-secret-syncer-fn8l9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt","openshift-image-registry/node-ca-sncd9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-bqhww","openshift-multus/network-metrics-daemon-l7j4q","kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-nrx4t","openshift-network-diagnostics/network-check-target-g2vhp","openshift-network-operator/iptables-alerter-5skrq","openshift-ovn-kubernetes/ovnkube-node-8ptts"] Nov 03 18:43:44.604593 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.604570 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:43:44.606731 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.606713 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:43:44.608905 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.608884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.611035 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.611020 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:44.611102 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:44.611076 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:43:44.613187 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.613170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.615480 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.615460 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:43:44.617798 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.617781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.620052 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.620036 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:44.620129 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:44.620089 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:43:44.622387 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.622369 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.624688 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.624671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:44.624769 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:44.624731 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:43:44.626997 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.626980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:43:44.629793 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.629778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.642757 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.642741 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 18:43:44.650016 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.649999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-log-socket\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.650103 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-cni-bin\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.650103 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650056 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcb52\" (UniqueName: \"kubernetes.io/projected/9a63bf8a-50a3-4068-9835-50cfd2156350-kube-api-access-vcb52\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.650184 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcd42898-f7cc-4198-851b-db51f7ba110a-kubelet-config\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:44.650184 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.650251 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-systemd-units\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.650251 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-run-netns\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.650251 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-etc-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.650340 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-cnibin\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.650340 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-conf-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.650340 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-tuned\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.650454 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-var-lib-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.650454 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-cni-bin\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.650454 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650396 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-socket-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.650454 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysctl-d\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.650583 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhtzm\" (UniqueName: \"kubernetes.io/projected/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-kube-api-access-fhtzm\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.650583 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-ovn\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.650583 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9126b699-4c1b-46ca-976c-a4e1813357a1-agent-certs\") pod \"konnectivity-agent-dwp5g\" (UID: \"9126b699-4c1b-46ca-976c-a4e1813357a1\") " pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:43:44.650583 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0169d01-fdae-40a1-ad57-8c380c023d61-hosts-file\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:43:44.650583 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650541 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcd42898-f7cc-4198-851b-db51f7ba110a-dbus\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:44.650583 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-host-slash\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:43:44.650761 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.650761 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.650761 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl646\" (UniqueName: \"kubernetes.io/projected/c7bddb76-6f22-430e-9016-ce19e959dd48-kube-api-access-gl646\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.650761 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-cni-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.650761 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650703 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-socket-dir-parent\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.650761 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.650761 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysconfig\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.650966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650775 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysctl-conf\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.650966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-sys\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.650966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-systemd\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.650966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9126b699-4c1b-46ca-976c-a4e1813357a1-konnectivity-ca\") pod \"konnectivity-agent-dwp5g\" (UID: \"9126b699-4c1b-46ca-976c-a4e1813357a1\") " pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:43:44.650966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-kubelet\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.650966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-cni-binary-copy\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.650966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-node-log\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.650966 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7bddb76-6f22-430e-9016-ce19e959dd48-ovn-node-metrics-cert\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.651193 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0169d01-fdae-40a1-ad57-8c380c023d61-tmp-dir\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:43:44.651193 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.650994 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-etc-kubernetes\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.651193 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z575t\" (UniqueName: \"kubernetes.io/projected/e1055486-a6b2-425f-9c20-89c93df7e080-kube-api-access-z575t\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.651193 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-system-cni-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.651193 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651098 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-sys-fs\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.651193 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv7j9\" (UniqueName: \"kubernetes.io/projected/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-kube-api-access-mv7j9\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.651193 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651142 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-run\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.651193 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.651193 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-netns\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.651467 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651207 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-daemon-config\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.651467 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-device-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.651467 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651241 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8m9t\" (UniqueName: \"kubernetes.io/projected/d52d4fd1-25a8-44d6-94d8-13757fe7790e-kube-api-access-l8m9t\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:43:44.651467 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-lib-modules\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.651467 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-cni-netd\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.651467 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-ovnkube-config\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.651467 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-k8s-cni-cncf-io\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.651467 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651349 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-cni-multus\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.651467 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:44.651467 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-tmp\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.651467 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-system-cni-dir\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.651824 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqpjs\" (UniqueName: \"kubernetes.io/projected/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-kube-api-access-bqpjs\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:43:44.651824 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a63bf8a-50a3-4068-9835-50cfd2156350-cni-binary-copy\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.651824 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651527 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-hostroot\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.651824 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651542 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d52d4fd1-25a8-44d6-94d8-13757fe7790e-host\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:43:44.651824 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:44.651824 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvp79\" (UniqueName: \"kubernetes.io/projected/de669e87-278f-48af-bab0-625698973092-kube-api-access-zvp79\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:44.651824 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651736 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-cnibin\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.651824 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-ovnkube-script-lib\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.651824 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-multus-certs\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.651824 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-kubernetes\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651828 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-systemd\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-var-lib-kubelet\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d52d4fd1-25a8-44d6-94d8-13757fe7790e-serviceca\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-iptables-alerter-script\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzh4l\" (UniqueName: \"kubernetes.io/projected/b0169d01-fdae-40a1-ad57-8c380c023d61-kube-api-access-zzh4l\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.651994 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-modprobe-d\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.652012 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9mbx\" (UniqueName: \"kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx\") pod \"network-check-target-g2vhp\" (UID: \"894d2520-9e05-4460-a37d-f3f8377a488b\") " pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.652032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-slash\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.652053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-env-overrides\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.652074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-registration-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.652105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.652097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-host\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.652530 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.652119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-kubelet\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.659446 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.659417 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 18:43:44.659446 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.659421 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k4rz5\"" Nov 03 18:43:44.659561 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.659421 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 18:43:44.660890 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.660873 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 18:43:44.661026 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.660994 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q46mt\"" Nov 03 18:43:44.661099 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.661079 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 18:43:44.662856 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.662829 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-97rln\"" Nov 03 18:43:44.662975 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.662958 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 18:43:44.663114 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.663100 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 18:43:44.663114 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.663111 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 18:43:44.663240 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.663134 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 18:43:44.667332 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.667315 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 18:43:44.667430 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.667340 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 18:43:44.667430 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.667321 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 18:43:44.667506 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.667430 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8n5dg\"" Nov 03 18:43:44.669463 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.669441 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 18:43:44.669693 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.669679 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45nsg\"" Nov 03 18:43:44.669749 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.669691 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 18:43:44.669749 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.669691 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 18:43:44.671926 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.671911 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 18:43:44.672004 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.671993 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 18:43:44.672161 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.672149 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jwgdr\"" Nov 03 18:43:44.678818 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.678795 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 18:43:44.707149 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.707123 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:43:44.707252 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.707151 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-grblk\"" Nov 03 18:43:44.711187 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.711169 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d8dtx\"" Nov 03 18:43:44.711278 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.711264 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qnhrd\"" Nov 03 18:43:44.711369 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.711358 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 18:43:44.711790 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.711773 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 18:43:44.711790 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.711785 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 18:43:44.711893 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.711796 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 18:43:44.711893 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.711803 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 18:43:44.711893 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.711879 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 18:43:44.713019 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.712996 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 18:43:44.713019 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.713005 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 18:43:44.713139 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.713005 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:43:44.713139 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.713088 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 18:43:44.713139 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.713133 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f8zq8" Nov 03 18:43:44.752517 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-socket-dir-parent\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.752623 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.752623 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysconfig\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.752623 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysctl-conf\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.752623 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-sys\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.752623 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-systemd\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.752623 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9126b699-4c1b-46ca-976c-a4e1813357a1-konnectivity-ca\") pod \"konnectivity-agent-dwp5g\" (UID: \"9126b699-4c1b-46ca-976c-a4e1813357a1\") " pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752622 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-socket-dir-parent\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-kubelet\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752659 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysconfig\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752679 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752695 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-sys\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752702 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-kubelet\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-cni-binary-copy\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-node-log\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752782 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-systemd\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7bddb76-6f22-430e-9016-ce19e959dd48-ovn-node-metrics-cert\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752800 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysctl-conf\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0169d01-fdae-40a1-ad57-8c380c023d61-tmp-dir\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752855 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-node-log\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-etc-kubernetes\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752889 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z575t\" (UniqueName: \"kubernetes.io/projected/e1055486-a6b2-425f-9c20-89c93df7e080-kube-api-access-z575t\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-system-cni-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.752969 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-sys-fs\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv7j9\" (UniqueName: \"kubernetes.io/projected/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-kube-api-access-mv7j9\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.752990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-run\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.753037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.753077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-netns\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.753078 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-system-cni-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.753081 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-sys-fs\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.753042 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-run\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.753166 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.753324 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9126b699-4c1b-46ca-976c-a4e1813357a1-konnectivity-ca\") pod \"konnectivity-agent-dwp5g\" (UID: \"9126b699-4c1b-46ca-976c-a4e1813357a1\") " pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.753326 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0169d01-fdae-40a1-ad57-8c380c023d61-tmp-dir\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.753633 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-cni-binary-copy\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.753647 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.754028 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-daemon-config\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-device-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8m9t\" (UniqueName: \"kubernetes.io/projected/d52d4fd1-25a8-44d6-94d8-13757fe7790e-kube-api-access-l8m9t\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-lib-modules\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-cni-netd\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754128 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-device-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-ovnkube-config\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-k8s-cni-cncf-io\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-cni-multus\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754205 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-cni-netd\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754237 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-lib-modules\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754269 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-k8s-cni-cncf-io\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-tmp\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-system-cni-dir\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqpjs\" (UniqueName: \"kubernetes.io/projected/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-kube-api-access-bqpjs\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754349 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-system-cni-dir\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.754691 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a63bf8a-50a3-4068-9835-50cfd2156350-cni-binary-copy\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-hostroot\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d52d4fd1-25a8-44d6-94d8-13757fe7790e-host\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvp79\" (UniqueName: \"kubernetes.io/projected/de669e87-278f-48af-bab0-625698973092-kube-api-access-zvp79\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-cnibin\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-ovnkube-script-lib\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-multus-certs\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-kubernetes\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-systemd\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-var-lib-kubelet\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754684 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-daemon-config\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d52d4fd1-25a8-44d6-94d8-13757fe7790e-serviceca\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-iptables-alerter-script\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754756 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d52d4fd1-25a8-44d6-94d8-13757fe7790e-host\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:43:44.755575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzh4l\" (UniqueName: \"kubernetes.io/projected/b0169d01-fdae-40a1-ad57-8c380c023d61-kube-api-access-zzh4l\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754806 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-hostroot\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754815 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-systemd\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:44.754808 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754865 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-multus-certs\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:44.754885 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs podName:de669e87-278f-48af-bab0-625698973092 nodeName:}" failed. No retries permitted until 2026-11-03 18:43:45.254865168 +0000 UTC m=+2.325895519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs") pod "network-metrics-daemon-l7j4q" (UID: "de669e87-278f-48af-bab0-625698973092") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754884 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-etc-kubernetes\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754820 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-modprobe-d\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754918 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-var-lib-kubelet\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mbx\" (UniqueName: \"kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx\") pod \"network-check-target-g2vhp\" (UID: \"894d2520-9e05-4460-a37d-f3f8377a488b\") " pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754936 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-cnibin\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754950 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a63bf8a-50a3-4068-9835-50cfd2156350-cni-binary-copy\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-slash\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754964 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-ovnkube-config\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.754995 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-slash\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-env-overrides\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755007 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-cni-multus\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.756452 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:44.755043 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755061 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755104 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-registration-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-host\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755195 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-kubelet\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-log-socket\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:44.755261 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret podName:dcd42898-f7cc-4198-851b-db51f7ba110a nodeName:}" failed. No retries permitted until 2026-11-03 18:43:45.255242291 +0000 UTC m=+2.326272642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret") pod "global-pull-secret-syncer-fn8l9" (UID: "dcd42898-f7cc-4198-851b-db51f7ba110a") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755273 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-netns\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-cni-bin\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcb52\" (UniqueName: \"kubernetes.io/projected/9a63bf8a-50a3-4068-9835-50cfd2156350-kube-api-access-vcb52\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755343 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-ovnkube-script-lib\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcd42898-f7cc-4198-851b-db51f7ba110a-kubelet-config\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755392 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcd42898-f7cc-4198-851b-db51f7ba110a-kubelet-config\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755553 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.757330 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-systemd-units\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755595 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-run-netns\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755109 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-kubernetes\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755652 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-registration-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755694 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-env-overrides\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-etc-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-cnibin\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-conf-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-tuned\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-var-lib-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-cni-bin\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755936 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d52d4fd1-25a8-44d6-94d8-13757fe7790e-serviceca\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-socket-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.755964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysctl-d\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhtzm\" (UniqueName: \"kubernetes.io/projected/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-kube-api-access-fhtzm\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-ovn\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756120 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-conf-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.757841 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9126b699-4c1b-46ca-976c-a4e1813357a1-agent-certs\") pod \"konnectivity-agent-dwp5g\" (UID: \"9126b699-4c1b-46ca-976c-a4e1813357a1\") " pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0169d01-fdae-40a1-ad57-8c380c023d61-hosts-file\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcd42898-f7cc-4198-851b-db51f7ba110a-dbus\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-host-slash\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756222 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysctl-d\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756286 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-modprobe-d\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756294 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-iptables-alerter-script\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl646\" (UniqueName: \"kubernetes.io/projected/c7bddb76-6f22-430e-9016-ce19e959dd48-kube-api-access-gl646\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756310 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-kubelet\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-cni-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756343 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-host\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756457 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756499 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-systemd-units\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756505 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-run-netns\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756539 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-cni-bin\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.758335 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756578 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-var-lib-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758875 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756593 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-log-socket\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758875 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756613 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-cni-bin\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758875 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756662 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-cnibin\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.758875 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756708 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-socket-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.758875 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756758 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758875 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756781 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0169d01-fdae-40a1-ad57-8c380c023d61-hosts-file\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:43:44.758875 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756796 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-host-slash\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:43:44.758875 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756815 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcd42898-f7cc-4198-851b-db51f7ba110a-dbus\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:44.758875 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756830 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-ovn\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758875 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756837 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-etc-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.758875 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.756848 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-cni-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.759310 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.758927 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-tmp\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.759310 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.758990 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7bddb76-6f22-430e-9016-ce19e959dd48-ovn-node-metrics-cert\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.759310 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.759021 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9126b699-4c1b-46ca-976c-a4e1813357a1-agent-certs\") pod \"konnectivity-agent-dwp5g\" (UID: \"9126b699-4c1b-46ca-976c-a4e1813357a1\") " pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:43:44.759310 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.759025 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-tuned\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.816556 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:44.816478 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:43:44.816556 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:44.816515 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:43:44.816556 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:44.816529 2579 projected.go:196] Error preparing data for projected volume kube-api-access-g9mbx for pod openshift-network-diagnostics/network-check-target-g2vhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:43:44.816836 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:44.816632 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx podName:894d2520-9e05-4460-a37d-f3f8377a488b nodeName:}" failed. No retries permitted until 2026-11-03 18:43:45.31661033 +0000 UTC m=+2.387640684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g9mbx" (UniqueName: "kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx") pod "network-check-target-g2vhp" (UID: "894d2520-9e05-4460-a37d-f3f8377a488b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:43:44.818623 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.818585 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqpjs\" (UniqueName: \"kubernetes.io/projected/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-kube-api-access-bqpjs\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:43:44.819562 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.819539 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z575t\" (UniqueName: \"kubernetes.io/projected/e1055486-a6b2-425f-9c20-89c93df7e080-kube-api-access-z575t\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:44.819562 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.819556 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvp79\" (UniqueName: \"kubernetes.io/projected/de669e87-278f-48af-bab0-625698973092-kube-api-access-zvp79\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:44.819673 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.819621 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8m9t\" (UniqueName: \"kubernetes.io/projected/d52d4fd1-25a8-44d6-94d8-13757fe7790e-kube-api-access-l8m9t\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:43:44.819673 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.819636 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv7j9\" (UniqueName: \"kubernetes.io/projected/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-kube-api-access-mv7j9\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:44.823017 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.822985 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzh4l\" (UniqueName: \"kubernetes.io/projected/b0169d01-fdae-40a1-ad57-8c380c023d61-kube-api-access-zzh4l\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:43:44.823138 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.823081 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcb52\" (UniqueName: \"kubernetes.io/projected/9a63bf8a-50a3-4068-9835-50cfd2156350-kube-api-access-vcb52\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.824021 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.824002 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl646\" (UniqueName: \"kubernetes.io/projected/c7bddb76-6f22-430e-9016-ce19e959dd48-kube-api-access-gl646\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:44.825064 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.825040 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhtzm\" (UniqueName: \"kubernetes.io/projected/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-kube-api-access-fhtzm\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:44.929539 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.929510 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:43:44.948675 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.946911 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:43:44.968530 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.968503 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rwwj5" Nov 03 18:43:44.988539 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:44.988515 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:43:45.008012 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.007988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:43:45.035635 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.035618 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:43:45.041705 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.041686 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:43:45.049655 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:45.049626 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1055486_a6b2_425f_9c20_89c93df7e080.slice/crio-4dfa67acd059b2489cdc964514150aefa2d7bb74d2c5599b662be0502804fb85 WatchSource:0}: Error finding container 4dfa67acd059b2489cdc964514150aefa2d7bb74d2c5599b662be0502804fb85: Status 404 returned error can't find the container with id 4dfa67acd059b2489cdc964514150aefa2d7bb74d2c5599b662be0502804fb85 Nov 03 18:43:45.050245 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:45.050224 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e9a2c3_c4e3_4e78_9157_926fae93f8a8.slice/crio-cc57b913b081983f0738901b4ba4e13009779804fb419fbe114fb30ed7acbe3c WatchSource:0}: Error finding container cc57b913b081983f0738901b4ba4e13009779804fb419fbe114fb30ed7acbe3c: Status 404 returned error can't find the container with id cc57b913b081983f0738901b4ba4e13009779804fb419fbe114fb30ed7acbe3c Nov 03 18:43:45.051011 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:45.050991 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52d4fd1_25a8_44d6_94d8_13757fe7790e.slice/crio-5f2127f149bf105f7e6d273f1d355b361c93bebb08b3d440900c100e29b5ce20 WatchSource:0}: Error finding container 5f2127f149bf105f7e6d273f1d355b361c93bebb08b3d440900c100e29b5ce20: Status 404 returned error can't find the container with id 5f2127f149bf105f7e6d273f1d355b361c93bebb08b3d440900c100e29b5ce20 Nov 03 18:43:45.051633 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:45.051613 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1997d138e76444593b76299593de1e7.slice/crio-2ad4518d7c2651fa29049a332906a048239f09fceb8a9dc1a41b5ca1e623ca97 WatchSource:0}: Error finding container 2ad4518d7c2651fa29049a332906a048239f09fceb8a9dc1a41b5ca1e623ca97: Status 404 returned error can't find the container with id 2ad4518d7c2651fa29049a332906a048239f09fceb8a9dc1a41b5ca1e623ca97 Nov 03 18:43:45.052258 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:45.052231 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9f7f968eecc0f1a776909177a87d50.slice/crio-13b91cb8dffc8ff97a59294067359465bdeea6fcc392e9e5c8a9542f7ae216f4 WatchSource:0}: Error finding container 13b91cb8dffc8ff97a59294067359465bdeea6fcc392e9e5c8a9542f7ae216f4: Status 404 returned error can't find the container with id 13b91cb8dffc8ff97a59294067359465bdeea6fcc392e9e5c8a9542f7ae216f4 Nov 03 18:43:45.054765 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.054747 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:43:45.056452 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:45.056430 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1e9ac2_4936_4d85_89ab_33cc3ef7a684.slice/crio-ac0e0424458eaec2e230be81545fa2bd38d8e37261fba1596afafdf33559456d WatchSource:0}: Error finding container ac0e0424458eaec2e230be81545fa2bd38d8e37261fba1596afafdf33559456d: Status 404 returned error can't find the container with id ac0e0424458eaec2e230be81545fa2bd38d8e37261fba1596afafdf33559456d Nov 03 18:43:45.057025 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.056987 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:43:45.064196 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:45.064175 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0f1aae_624b_4735_ae9b_d27b716a6d9c.slice/crio-9160a39d465ef69ed863605fb4c1f3319a5d102edab38c03ae17f3fba0ebf194 WatchSource:0}: Error finding container 9160a39d465ef69ed863605fb4c1f3319a5d102edab38c03ae17f3fba0ebf194: Status 404 returned error can't find the container with id 9160a39d465ef69ed863605fb4c1f3319a5d102edab38c03ae17f3fba0ebf194 Nov 03 18:43:45.078858 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.078841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:43:45.087739 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:45.087716 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bddb76_6f22_430e_9016_ce19e959dd48.slice/crio-512f5c3d2b95a5f3a86a58b3129b890070fb21e8b90bd51f1825693e089205de WatchSource:0}: Error finding container 512f5c3d2b95a5f3a86a58b3129b890070fb21e8b90bd51f1825693e089205de: Status 404 returned error can't find the container with id 512f5c3d2b95a5f3a86a58b3129b890070fb21e8b90bd51f1825693e089205de Nov 03 18:43:45.232315 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.232291 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:43:45.260232 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.260210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:45.260339 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.260241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:45.260393 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:45.260378 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:43:45.260472 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:45.260437 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret podName:dcd42898-f7cc-4198-851b-db51f7ba110a nodeName:}" failed. No retries permitted until 2026-11-03 18:43:46.260421451 +0000 UTC m=+3.331451797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret") pod "global-pull-secret-syncer-fn8l9" (UID: "dcd42898-f7cc-4198-851b-db51f7ba110a") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:43:45.260472 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:45.260377 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:43:45.260545 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:45.260473 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs podName:de669e87-278f-48af-bab0-625698973092 nodeName:}" failed. No retries permitted until 2026-11-03 18:43:46.260462654 +0000 UTC m=+3.331492986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs") pod "network-metrics-daemon-l7j4q" (UID: "de669e87-278f-48af-bab0-625698973092") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:43:45.347497 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.347420 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:43:45.363458 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.361740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mbx\" (UniqueName: \"kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx\") pod \"network-check-target-g2vhp\" (UID: \"894d2520-9e05-4460-a37d-f3f8377a488b\") " pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:45.363458 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:45.362006 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:43:45.363458 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:45.362027 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:43:45.363458 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:45.362041 2579 projected.go:196] Error preparing data for projected volume kube-api-access-g9mbx for pod openshift-network-diagnostics/network-check-target-g2vhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:43:45.363458 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:45.362105 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx podName:894d2520-9e05-4460-a37d-f3f8377a488b nodeName:}" failed. No retries permitted until 2026-11-03 18:43:46.362086233 +0000 UTC m=+3.433116573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g9mbx" (UniqueName: "kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx") pod "network-check-target-g2vhp" (UID: "894d2520-9e05-4460-a37d-f3f8377a488b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:43:45.403155 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.403130 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:43:45.415665 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.415624 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" event={"ID":"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684","Type":"ContainerStarted","Data":"ac0e0424458eaec2e230be81545fa2bd38d8e37261fba1596afafdf33559456d"} Nov 03 18:43:45.416713 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.416693 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bqhww" event={"ID":"e1055486-a6b2-425f-9c20-89c93df7e080","Type":"ContainerStarted","Data":"4dfa67acd059b2489cdc964514150aefa2d7bb74d2c5599b662be0502804fb85"} Nov 03 18:43:45.417700 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.417671 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" event={"ID":"b1997d138e76444593b76299593de1e7","Type":"ContainerStarted","Data":"2ad4518d7c2651fa29049a332906a048239f09fceb8a9dc1a41b5ca1e623ca97"} Nov 03 18:43:45.418727 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.418708 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" event={"ID":"9d9f7f968eecc0f1a776909177a87d50","Type":"ContainerStarted","Data":"13b91cb8dffc8ff97a59294067359465bdeea6fcc392e9e5c8a9542f7ae216f4"} Nov 03 18:43:45.419864 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.419845 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" event={"ID":"16e9a2c3-c4e3-4e78-9157-926fae93f8a8","Type":"ContainerStarted","Data":"cc57b913b081983f0738901b4ba4e13009779804fb419fbe114fb30ed7acbe3c"} Nov 03 18:43:45.420905 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.420886 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sncd9" event={"ID":"d52d4fd1-25a8-44d6-94d8-13757fe7790e","Type":"ContainerStarted","Data":"5f2127f149bf105f7e6d273f1d355b361c93bebb08b3d440900c100e29b5ce20"} Nov 03 18:43:45.421874 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.421857 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" event={"ID":"c7bddb76-6f22-430e-9016-ce19e959dd48","Type":"ContainerStarted","Data":"512f5c3d2b95a5f3a86a58b3129b890070fb21e8b90bd51f1825693e089205de"} Nov 03 18:43:45.422877 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.422859 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5skrq" event={"ID":"0e0f1aae-624b-4735-ae9b-d27b716a6d9c","Type":"ContainerStarted","Data":"9160a39d465ef69ed863605fb4c1f3319a5d102edab38c03ae17f3fba0ebf194"} Nov 03 18:43:45.716605 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.716551 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:38:44 +0000 UTC" deadline="2028-05-13 05:52:30.35605616 +0000 UTC" Nov 03 18:43:45.716605 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:45.716597 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13355h8m44.639461444s" Nov 03 18:43:45.802763 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:45.802736 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9126b699_4c1b_46ca_976c_a4e1813357a1.slice/crio-c5688730c6f24ee8bcc7d9d40f40fc26f099049f14af184f4c1a9add2b5ebcf1 WatchSource:0}: Error finding container c5688730c6f24ee8bcc7d9d40f40fc26f099049f14af184f4c1a9add2b5ebcf1: Status 404 returned error can't find the container with id c5688730c6f24ee8bcc7d9d40f40fc26f099049f14af184f4c1a9add2b5ebcf1 Nov 03 18:43:45.861567 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:45.861541 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0169d01_fdae_40a1_ad57_8c380c023d61.slice/crio-33f0f81883389e4be510096c5e425b5cafaa34c1a94a88f4b72aa7f2043b18d5 WatchSource:0}: Error finding container 33f0f81883389e4be510096c5e425b5cafaa34c1a94a88f4b72aa7f2043b18d5: Status 404 returned error can't find the container with id 33f0f81883389e4be510096c5e425b5cafaa34c1a94a88f4b72aa7f2043b18d5 Nov 03 18:43:45.911205 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:43:45.911182 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a63bf8a_50a3_4068_9835_50cfd2156350.slice/crio-f0205d6051b2957d46025f61c2823434f601e946565b2248eb4caf1af9641af5 WatchSource:0}: Error finding container f0205d6051b2957d46025f61c2823434f601e946565b2248eb4caf1af9641af5: Status 404 returned error can't find the container with id f0205d6051b2957d46025f61c2823434f601e946565b2248eb4caf1af9641af5 Nov 03 18:43:46.270340 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:46.270295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:46.270522 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:46.270360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:46.270522 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:46.270515 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:43:46.270640 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:46.270570 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs podName:de669e87-278f-48af-bab0-625698973092 nodeName:}" failed. No retries permitted until 2026-11-03 18:43:48.270553007 +0000 UTC m=+5.341583352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs") pod "network-metrics-daemon-l7j4q" (UID: "de669e87-278f-48af-bab0-625698973092") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:43:46.270640 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:46.270637 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:43:46.270751 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:46.270669 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret podName:dcd42898-f7cc-4198-851b-db51f7ba110a nodeName:}" failed. No retries permitted until 2026-11-03 18:43:48.270658156 +0000 UTC m=+5.341688502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret") pod "global-pull-secret-syncer-fn8l9" (UID: "dcd42898-f7cc-4198-851b-db51f7ba110a") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:43:46.370956 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:46.370669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mbx\" (UniqueName: \"kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx\") pod \"network-check-target-g2vhp\" (UID: \"894d2520-9e05-4460-a37d-f3f8377a488b\") " pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:46.370956 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:46.370835 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:43:46.370956 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:46.370856 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:43:46.370956 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:46.370869 2579 projected.go:196] Error preparing data for projected volume kube-api-access-g9mbx for pod openshift-network-diagnostics/network-check-target-g2vhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:43:46.370956 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:46.370922 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx podName:894d2520-9e05-4460-a37d-f3f8377a488b nodeName:}" failed. No retries permitted until 2026-11-03 18:43:48.370905132 +0000 UTC m=+5.441935468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g9mbx" (UniqueName: "kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx") pod "network-check-target-g2vhp" (UID: "894d2520-9e05-4460-a37d-f3f8377a488b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:43:46.412735 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:46.412250 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:46.412735 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:46.412396 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:43:46.412735 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:46.412485 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:46.412735 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:46.412575 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:43:46.412735 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:46.412634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:46.412735 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:46.412693 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:43:46.438398 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:46.438364 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dwp5g" event={"ID":"9126b699-4c1b-46ca-976c-a4e1813357a1","Type":"ContainerStarted","Data":"c5688730c6f24ee8bcc7d9d40f40fc26f099049f14af184f4c1a9add2b5ebcf1"} Nov 03 18:43:46.445676 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:46.445623 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rwwj5" event={"ID":"9a63bf8a-50a3-4068-9835-50cfd2156350","Type":"ContainerStarted","Data":"f0205d6051b2957d46025f61c2823434f601e946565b2248eb4caf1af9641af5"} Nov 03 18:43:46.451732 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:46.451681 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dlbxv" event={"ID":"b0169d01-fdae-40a1-ad57-8c380c023d61","Type":"ContainerStarted","Data":"33f0f81883389e4be510096c5e425b5cafaa34c1a94a88f4b72aa7f2043b18d5"} Nov 03 18:43:46.717929 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:46.717636 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:38:44 +0000 UTC" deadline="2028-05-31 06:01:26.176548266 +0000 UTC" Nov 03 18:43:46.717929 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:46.717675 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13787h17m39.458878125s" Nov 03 18:43:48.286440 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:48.286403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:48.287041 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:48.286451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:48.291548 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:48.291503 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:43:48.291705 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:48.291613 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret podName:dcd42898-f7cc-4198-851b-db51f7ba110a nodeName:}" failed. No retries permitted until 2026-11-03 18:43:52.291591408 +0000 UTC m=+9.362621743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret") pod "global-pull-secret-syncer-fn8l9" (UID: "dcd42898-f7cc-4198-851b-db51f7ba110a") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:43:48.292008 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:48.291983 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:43:48.292076 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:48.292070 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs podName:de669e87-278f-48af-bab0-625698973092 nodeName:}" failed. No retries permitted until 2026-11-03 18:43:52.292051213 +0000 UTC m=+9.363081558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs") pod "network-metrics-daemon-l7j4q" (UID: "de669e87-278f-48af-bab0-625698973092") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:43:48.387301 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:48.387255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mbx\" (UniqueName: \"kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx\") pod \"network-check-target-g2vhp\" (UID: \"894d2520-9e05-4460-a37d-f3f8377a488b\") " pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:48.387526 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:48.387507 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:43:48.387622 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:48.387538 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:43:48.387622 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:48.387550 2579 projected.go:196] Error preparing data for projected volume kube-api-access-g9mbx for pod openshift-network-diagnostics/network-check-target-g2vhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:43:48.387622 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:48.387608 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx podName:894d2520-9e05-4460-a37d-f3f8377a488b nodeName:}" failed. No retries permitted until 2026-11-03 18:43:52.387592146 +0000 UTC m=+9.458622494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g9mbx" (UniqueName: "kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx") pod "network-check-target-g2vhp" (UID: "894d2520-9e05-4460-a37d-f3f8377a488b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:43:48.412424 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:48.412396 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:48.412584 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:48.412504 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:43:48.412584 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:48.412556 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:48.412690 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:48.412612 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:43:48.412690 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:48.412658 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:48.412794 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:48.412704 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:43:49.501437 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:49.501318 2579 generic.go:356] "Generic (PLEG): container finished" podID="e1055486-a6b2-425f-9c20-89c93df7e080" containerID="1ef10b4e12be1f840c85e2a20474546438b93143d1993e8bd37e6ad13fb2c6ed" exitCode=0 Nov 03 18:43:49.501437 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:49.501403 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bqhww" event={"ID":"e1055486-a6b2-425f-9c20-89c93df7e080","Type":"ContainerDied","Data":"1ef10b4e12be1f840c85e2a20474546438b93143d1993e8bd37e6ad13fb2c6ed"} Nov 03 18:43:50.412819 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:50.412076 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:50.412819 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:50.412472 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:43:50.412819 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:50.412565 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:50.412819 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:50.412643 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:43:50.412819 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:50.412695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:50.412819 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:50.412776 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:43:50.514732 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:50.514693 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" event={"ID":"b1997d138e76444593b76299593de1e7","Type":"ContainerStarted","Data":"640b23f4b271b3f02d9804c1d7bd812092326f8cd39c1f801e2efcc6d8b8b3cb"} Nov 03 18:43:50.518697 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:50.518648 2579 generic.go:356] "Generic (PLEG): container finished" podID="9d9f7f968eecc0f1a776909177a87d50" containerID="47e1459aae180e6d1bffcf31586589104d81c833bac81511f6401f69712dca4c" exitCode=0 Nov 03 18:43:50.518697 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:50.518695 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" event={"ID":"9d9f7f968eecc0f1a776909177a87d50","Type":"ContainerDied","Data":"47e1459aae180e6d1bffcf31586589104d81c833bac81511f6401f69712dca4c"} Nov 03 18:43:50.629062 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:50.627873 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" podStartSLOduration=6.627855082 podStartE2EDuration="6.627855082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:50.627632893 +0000 UTC m=+7.698663245" watchObservedRunningTime="2026-11-03 18:43:50.627855082 +0000 UTC m=+7.698885435" Nov 03 18:43:51.525229 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:51.525198 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" event={"ID":"9d9f7f968eecc0f1a776909177a87d50","Type":"ContainerStarted","Data":"05f20bb96bfc278e55942eb8ce315ff88fae7fda1963c322e95ecf2a57f789bf"} Nov 03 18:43:51.530752 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:51.530722 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dwp5g" event={"ID":"9126b699-4c1b-46ca-976c-a4e1813357a1","Type":"ContainerStarted","Data":"32a9313e5c7ad9479997e5ad128a097df1e97cfb88e835c4392935d368a15d10"} Nov 03 18:43:51.534397 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:51.534372 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sncd9" event={"ID":"d52d4fd1-25a8-44d6-94d8-13757fe7790e","Type":"ContainerStarted","Data":"1003ede9dee7ffea3a2ecd2bcebb4df45beffe6a479478af24274f1b859bab3e"} Nov 03 18:43:51.637565 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:51.637006 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" podStartSLOduration=7.636988985 podStartE2EDuration="7.636988985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:51.636394873 +0000 UTC m=+8.707425227" watchObservedRunningTime="2026-11-03 18:43:51.636988985 +0000 UTC m=+8.708019339" Nov 03 18:43:51.859590 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:51.858829 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dwp5g" podStartSLOduration=3.122579618 podStartE2EDuration="7.858808038s" totalImagesPullingTime="4.73622842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:51.748137818 +0000 UTC m=+8.819168173" watchObservedRunningTime="2026-11-03 18:43:51.858808038 +0000 UTC m=+8.929838394" Nov 03 18:43:51.859590 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:51.859197 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sncd9" podStartSLOduration=1.906302514 podStartE2EDuration="7.85918784s" totalImagesPullingTime="5.952885326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:51.858447464 +0000 UTC m=+8.929477817" watchObservedRunningTime="2026-11-03 18:43:51.85918784 +0000 UTC m=+8.930218194" Nov 03 18:43:52.327561 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:52.327207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:52.327561 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:52.327253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:52.327561 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:52.327382 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:43:52.327561 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:52.327438 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs podName:de669e87-278f-48af-bab0-625698973092 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:00.327421671 +0000 UTC m=+17.398452008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs") pod "network-metrics-daemon-l7j4q" (UID: "de669e87-278f-48af-bab0-625698973092") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:43:52.327561 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:52.327499 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:43:52.327561 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:52.327530 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret podName:dcd42898-f7cc-4198-851b-db51f7ba110a nodeName:}" failed. No retries permitted until 2026-11-03 18:44:00.327519252 +0000 UTC m=+17.398549585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret") pod "global-pull-secret-syncer-fn8l9" (UID: "dcd42898-f7cc-4198-851b-db51f7ba110a") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:43:52.412670 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:52.412237 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:52.412670 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:52.412393 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:43:52.412670 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:52.412454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:52.412670 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:52.412518 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:43:52.412670 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:52.412560 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:52.412670 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:52.412603 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:43:52.427807 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:52.427557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mbx\" (UniqueName: \"kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx\") pod \"network-check-target-g2vhp\" (UID: \"894d2520-9e05-4460-a37d-f3f8377a488b\") " pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:52.427807 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:52.427708 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:43:52.427807 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:52.427721 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:43:52.427807 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:52.427729 2579 projected.go:196] Error preparing data for projected volume kube-api-access-g9mbx for pod openshift-network-diagnostics/network-check-target-g2vhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:43:52.427807 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:52.427766 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx podName:894d2520-9e05-4460-a37d-f3f8377a488b nodeName:}" failed. No retries permitted until 2026-11-03 18:44:00.427754709 +0000 UTC m=+17.498785038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g9mbx" (UniqueName: "kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx") pod "network-check-target-g2vhp" (UID: "894d2520-9e05-4460-a37d-f3f8377a488b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:43:52.538704 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:52.537949 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" event={"ID":"16e9a2c3-c4e3-4e78-9157-926fae93f8a8","Type":"ContainerStarted","Data":"0795ddae385d98df7544a53ab54c61774e2e51864ea9373a4215503d73f79c8b"} Nov 03 18:43:53.981819 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:53.981767 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 18:43:54.412922 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:54.412606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:54.412922 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:54.412722 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:43:54.412922 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:54.412771 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:54.412922 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:54.412816 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:43:54.412922 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:54.412855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:54.412922 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:54.412899 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:43:54.418537 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:54.418446 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T18:43:53.981791708Z","UUID":"f053bcce-e43a-4531-8d3e-27bf3212a965","Handler":null,"Name":"","Endpoint":""} Nov 03 18:43:54.420013 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:54.419996 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 18:43:54.420111 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:54.420021 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 18:43:54.532265 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:54.532231 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:43:54.532817 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:54.532798 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:43:54.546863 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:54.546826 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rwwj5" event={"ID":"9a63bf8a-50a3-4068-9835-50cfd2156350","Type":"ContainerStarted","Data":"03af1f43da10f27ba6bf38e7a95249315223f101d0ae5e8044496093c157a9d9"} Nov 03 18:43:54.548395 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:54.548356 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dlbxv" event={"ID":"b0169d01-fdae-40a1-ad57-8c380c023d61","Type":"ContainerStarted","Data":"6728d0ebcf49bf12d6f9f87ea60c784d3ca500dbfbefed5a0cfdc0e7e17a7d0f"} Nov 03 18:43:54.550702 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:54.550674 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" event={"ID":"16e9a2c3-c4e3-4e78-9157-926fae93f8a8","Type":"ContainerStarted","Data":"48caf2a2a72e727ff23333bb8c262c9fcc6a043785b0944b134705cb92409eca"} Nov 03 18:43:54.754553 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:54.754053 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-dlbxv" podStartSLOduration=2.298126316 podStartE2EDuration="10.754034837s" totalImagesPullingTime="8.455908521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:54.753697502 +0000 UTC m=+11.824727860" watchObservedRunningTime="2026-11-03 18:43:54.754034837 +0000 UTC m=+11.825065191" Nov 03 18:43:55.554463 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:55.554431 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5skrq" event={"ID":"0e0f1aae-624b-4735-ae9b-d27b716a6d9c","Type":"ContainerStarted","Data":"5f1a3a6f34383cf6c5fc72d012fd921d83674d3870335a2f18cad1bfe6f0a304"} Nov 03 18:43:55.557403 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:55.557374 2579 generic.go:356] "Generic (PLEG): container finished" podID="e1055486-a6b2-425f-9c20-89c93df7e080" containerID="775f0f26382e3328a94b856d8408736a8a118092339c2eb791066c32c34ad254" exitCode=0 Nov 03 18:43:55.557765 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:55.557449 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bqhww" event={"ID":"e1055486-a6b2-425f-9c20-89c93df7e080","Type":"ContainerDied","Data":"775f0f26382e3328a94b856d8408736a8a118092339c2eb791066c32c34ad254"} Nov 03 18:43:55.665347 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:55.665090 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rwwj5" podStartSLOduration=3.749279452 podStartE2EDuration="11.665073125s" totalImagesPullingTime="7.915793673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:54.870070456 +0000 UTC m=+11.941100809" watchObservedRunningTime="2026-11-03 18:43:55.665073125 +0000 UTC m=+12.736103477" Nov 03 18:43:55.666027 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:55.665989 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5skrq" podStartSLOduration=3.194548428 podStartE2EDuration="11.665974676s" totalImagesPullingTime="8.471426248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:55.664986102 +0000 UTC m=+12.736016457" watchObservedRunningTime="2026-11-03 18:43:55.665974676 +0000 UTC m=+12.737005029" Nov 03 18:43:56.412800 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:56.412734 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:56.412961 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:56.412744 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:56.412961 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:56.412840 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:43:56.412961 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:56.412744 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:56.412961 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:56.412904 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:43:56.413104 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:56.412973 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:43:56.560876 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:56.560831 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" event={"ID":"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684","Type":"ContainerStarted","Data":"695e9954967cb378c059b76446398fd1368583caf2a50dcd9f85af15a60ce192"} Nov 03 18:43:56.563890 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:56.563864 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" event={"ID":"16e9a2c3-c4e3-4e78-9157-926fae93f8a8","Type":"ContainerStarted","Data":"da5528a6d7aa5d1a6115318d70a36cdb645cc92eb077a4e5afa1f99aef5c3be2"} Nov 03 18:43:56.674872 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:56.674814 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" podStartSLOduration=1.74012057 podStartE2EDuration="12.674796852s" totalImagesPullingTime="10.934676282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:56.674439238 +0000 UTC m=+13.745469593" watchObservedRunningTime="2026-11-03 18:43:56.674796852 +0000 UTC m=+13.745827208" Nov 03 18:43:56.788809 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:56.788748 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" podStartSLOduration=2.697281257 podStartE2EDuration="12.788727166s" totalImagesPullingTime="10.091445909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:56.78820676 +0000 UTC m=+13.859237114" watchObservedRunningTime="2026-11-03 18:43:56.788727166 +0000 UTC m=+13.859757522" Nov 03 18:43:57.568063 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:57.568026 2579 generic.go:356] "Generic (PLEG): container finished" podID="e1055486-a6b2-425f-9c20-89c93df7e080" containerID="f3cef8352fd8e6b081950409dc24d1331457e86e14f87701998ace59ed33bd58" exitCode=0 Nov 03 18:43:57.568856 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:57.568112 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bqhww" event={"ID":"e1055486-a6b2-425f-9c20-89c93df7e080","Type":"ContainerDied","Data":"f3cef8352fd8e6b081950409dc24d1331457e86e14f87701998ace59ed33bd58"} Nov 03 18:43:58.412279 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:58.412238 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:43:58.412437 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:58.412372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:43:58.412503 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:58.412483 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:43:58.412562 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:58.412541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:43:58.412699 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:58.412668 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:43:58.412805 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:43:58.412784 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:43:59.575370 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:59.575153 2579 generic.go:356] "Generic (PLEG): container finished" podID="e1055486-a6b2-425f-9c20-89c93df7e080" containerID="feacae9836a9d43444a0791e81ca7dc482383a8d8b49df3d0b0cc10577743294" exitCode=0 Nov 03 18:43:59.575370 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:43:59.575236 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bqhww" event={"ID":"e1055486-a6b2-425f-9c20-89c93df7e080","Type":"ContainerDied","Data":"feacae9836a9d43444a0791e81ca7dc482383a8d8b49df3d0b0cc10577743294"} Nov 03 18:44:00.386930 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:00.386894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:00.387126 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:00.386946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:00.387126 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:00.387067 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:00.387126 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:00.387088 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:00.387254 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:00.387151 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret podName:dcd42898-f7cc-4198-851b-db51f7ba110a nodeName:}" failed. No retries permitted until 2026-11-03 18:44:16.387130897 +0000 UTC m=+33.458161241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret") pod "global-pull-secret-syncer-fn8l9" (UID: "dcd42898-f7cc-4198-851b-db51f7ba110a") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:44:00.387254 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:00.387173 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs podName:de669e87-278f-48af-bab0-625698973092 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:16.387162998 +0000 UTC m=+33.458193343 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs") pod "network-metrics-daemon-l7j4q" (UID: "de669e87-278f-48af-bab0-625698973092") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:44:00.412009 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:00.411982 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:00.412009 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:00.412004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:00.412205 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:00.412087 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:00.412205 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:00.412094 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:44:00.412205 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:00.412154 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:44:00.412380 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:00.412231 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:44:00.487824 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:00.487789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mbx\" (UniqueName: \"kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx\") pod \"network-check-target-g2vhp\" (UID: \"894d2520-9e05-4460-a37d-f3f8377a488b\") " pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:00.487974 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:00.487937 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:44:00.487974 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:00.487957 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:44:00.487974 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:00.487970 2579 projected.go:196] Error preparing data for projected volume kube-api-access-g9mbx for pod openshift-network-diagnostics/network-check-target-g2vhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:00.488117 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:00.488035 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx podName:894d2520-9e05-4460-a37d-f3f8377a488b nodeName:}" failed. No retries permitted until 2026-11-03 18:44:16.488017511 +0000 UTC m=+33.559047854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g9mbx" (UniqueName: "kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx") pod "network-check-target-g2vhp" (UID: "894d2520-9e05-4460-a37d-f3f8377a488b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:44:01.582588 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:01.582561 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" event={"ID":"c7bddb76-6f22-430e-9016-ce19e959dd48","Type":"ContainerStarted","Data":"05d05e91ae3216767c550e4cbac44bc4de178ebde7ec92e5c2c3413bf7df5bdc"} Nov 03 18:44:01.583040 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:01.582749 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" event={"ID":"c7bddb76-6f22-430e-9016-ce19e959dd48","Type":"ContainerStarted","Data":"b5dcb3135417b22402a6c25a1b172591576791f32f15a59fb7fa0f91a4277589"} Nov 03 18:44:01.583040 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:01.582804 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" event={"ID":"c7bddb76-6f22-430e-9016-ce19e959dd48","Type":"ContainerStarted","Data":"f25f12761e65a0016950653f19a6f1c550b95ffa1a572052cca2e1b589bae873"} Nov 03 18:44:01.583040 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:01.582855 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" event={"ID":"c7bddb76-6f22-430e-9016-ce19e959dd48","Type":"ContainerStarted","Data":"27edaff4f5ffeea8d983123a573f88ec40060a1da9613fdc3f12ba4263d3f22d"} Nov 03 18:44:02.412767 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:02.412601 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:02.412883 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:02.412641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:02.412883 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:02.412854 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:44:02.412965 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:02.412919 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:44:02.412965 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:02.412654 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:02.413034 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:02.412989 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:44:02.587357 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:02.587265 2579 generic.go:356] "Generic (PLEG): container finished" podID="e1055486-a6b2-425f-9c20-89c93df7e080" containerID="2201b31a5663c0161077bfd7fae9439d237fca8e6ff0bdad1ae184e872076e9c" exitCode=0 Nov 03 18:44:02.587787 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:02.587359 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bqhww" event={"ID":"e1055486-a6b2-425f-9c20-89c93df7e080","Type":"ContainerDied","Data":"2201b31a5663c0161077bfd7fae9439d237fca8e6ff0bdad1ae184e872076e9c"} Nov 03 18:44:02.591479 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:02.590619 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" event={"ID":"c7bddb76-6f22-430e-9016-ce19e959dd48","Type":"ContainerStarted","Data":"fa3a5f9859c7359e22ea64e583307eeb0ffc4a73a466b8f3337c0aab050c8853"} Nov 03 18:44:02.591479 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:02.590677 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" event={"ID":"c7bddb76-6f22-430e-9016-ce19e959dd48","Type":"ContainerStarted","Data":"e52e2d775232008142ba310dfcdd680bbefee08dc6f54d709d1a4797dcdebc96"} Nov 03 18:44:03.595567 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:03.595536 2579 generic.go:356] "Generic (PLEG): container finished" podID="e1055486-a6b2-425f-9c20-89c93df7e080" containerID="33cc98e415c3c2a1109e54920f619cad45a89921bf3e9b70678782db06925b19" exitCode=0 Nov 03 18:44:03.596022 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:03.595575 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bqhww" event={"ID":"e1055486-a6b2-425f-9c20-89c93df7e080","Type":"ContainerDied","Data":"33cc98e415c3c2a1109e54920f619cad45a89921bf3e9b70678782db06925b19"} Nov 03 18:44:04.412509 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:04.412472 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:04.412652 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:04.412484 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:04.412652 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:04.412583 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:04.412652 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:04.412584 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:44:04.412772 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:04.412652 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:44:04.412772 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:04.412712 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:44:04.601006 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:04.600975 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bqhww" event={"ID":"e1055486-a6b2-425f-9c20-89c93df7e080","Type":"ContainerStarted","Data":"817cda739a56cfe433e9f4a814a3af961e13358ef5b7f353ebc8e734f13ad5f4"} Nov 03 18:44:04.604064 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:04.604044 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" event={"ID":"c7bddb76-6f22-430e-9016-ce19e959dd48","Type":"ContainerStarted","Data":"9a161a84ef1004279c34793545c8a366a51c622411d39bf30802801691da33fd"} Nov 03 18:44:04.814645 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:04.814600 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bqhww" podStartSLOduration=6.64478793 podStartE2EDuration="20.814587531s" totalImagesPullingTime="14.169799601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:04.814061185 +0000 UTC m=+21.885091537" watchObservedRunningTime="2026-11-03 18:44:04.814587531 +0000 UTC m=+21.885617882" Nov 03 18:44:06.412278 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:06.412243 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:06.412911 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:06.412343 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:06.412911 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:06.412365 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:06.412911 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:06.412442 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:44:06.412911 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:06.412503 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:44:06.412911 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:06.412580 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:44:06.610781 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:06.610741 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" event={"ID":"c7bddb76-6f22-430e-9016-ce19e959dd48","Type":"ContainerStarted","Data":"5fa21d8648a63499ad622f2827d59db680a6154ccdc3797ce221a58f515eeb6b"} Nov 03 18:44:06.611121 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:06.611093 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:06.626285 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:06.626123 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:06.879212 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:06.879162 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" podStartSLOduration=6.6551035160000005 podStartE2EDuration="22.879149339s" totalImagesPullingTime="16.224045823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:06.855860337 +0000 UTC m=+23.926890684" watchObservedRunningTime="2026-11-03 18:44:06.879149339 +0000 UTC m=+23.950179690" Nov 03 18:44:07.613606 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:07.613578 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:07.614053 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:07.613663 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:07.627714 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:07.627691 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:08.411988 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:08.411955 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:08.412129 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:08.411955 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:08.412129 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:08.412077 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:44:08.412129 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:08.411960 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:08.412249 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:08.412171 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:44:08.412249 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:08.412206 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:44:08.540766 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:08.540736 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fn8l9"] Nov 03 18:44:08.543410 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:08.543386 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g2vhp"] Nov 03 18:44:08.546391 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:08.546369 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l7j4q"] Nov 03 18:44:08.615371 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:08.615343 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:08.615786 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:08.615352 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:08.615786 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:08.615471 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:44:08.615786 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:08.615503 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:44:08.615786 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:08.615551 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:08.615786 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:08.615598 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:44:10.411752 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:10.411724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:10.412105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:10.411817 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:10.412105 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:10.411826 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:44:10.412105 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:10.411903 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:44:10.412105 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:10.411940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:10.412105 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:10.411999 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:44:11.770448 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:11.770422 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:44:11.770874 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:11.770559 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:44:11.770977 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:11.770957 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:44:12.412609 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:12.412578 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:12.412774 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:12.412644 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:12.412774 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:12.412667 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:12.412774 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:12.412753 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2vhp" podUID="894d2520-9e05-4460-a37d-f3f8377a488b" Nov 03 18:44:12.412890 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:12.412794 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7j4q" podUID="de669e87-278f-48af-bab0-625698973092" Nov 03 18:44:12.412890 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:12.412848 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fn8l9" podUID="dcd42898-f7cc-4198-851b-db51f7ba110a" Nov 03 18:44:13.792252 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.792223 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeReady" Nov 03 18:44:13.792605 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.792317 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 18:44:13.903450 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.903418 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gjwzc"] Nov 03 18:44:13.911963 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.911942 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gjwzc" Nov 03 18:44:13.951773 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.951738 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ht2s6"] Nov 03 18:44:13.956065 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.956045 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l6xkx"] Nov 03 18:44:13.956195 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.956156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ht2s6" Nov 03 18:44:13.960048 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.960031 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-plzjk"] Nov 03 18:44:13.960154 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.960135 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l6xkx" Nov 03 18:44:13.963423 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.963311 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78c7cd5678-n2phb"] Nov 03 18:44:13.963423 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.963417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:13.967100 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.967085 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67986d9d7f-m6wxv"] Nov 03 18:44:13.967195 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.967181 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:13.972542 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.972522 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx"] Nov 03 18:44:13.972644 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.972629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:13.975891 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.975873 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8dd798ffc-wswht"] Nov 03 18:44:13.975977 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.975963 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:13.979420 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.979404 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722"] Nov 03 18:44:13.979524 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.979509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:13.982747 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.982729 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:13.987686 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.987665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hd86\" (UniqueName: \"kubernetes.io/projected/48738200-5d6d-4a3e-8798-700496cd1681-kube-api-access-2hd86\") pod \"kas-connection-checker-685895c7c9-gjwzc\" (UID: \"48738200-5d6d-4a3e-8798-700496cd1681\") " pod="kube-system/kas-connection-checker-685895c7c9-gjwzc" Nov 03 18:44:13.999647 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.999627 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 18:44:13.999647 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:13.999637 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 18:44:14.000146 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.000132 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4"] Nov 03 18:44:14.003262 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.003247 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-48wh4"] Nov 03 18:44:14.003382 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.003368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4" Nov 03 18:44:14.007364 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.007345 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg"] Nov 03 18:44:14.007443 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.007408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.011179 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.011161 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4"] Nov 03 18:44:14.011280 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.011214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:14.014224 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.014206 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9q8tt"] Nov 03 18:44:14.014398 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.014378 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:14.014697 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.014676 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 18:44:14.016834 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.016817 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t"] Nov 03 18:44:14.016938 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.016925 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:14.019776 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.019742 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9pzjb"] Nov 03 18:44:14.019893 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.019861 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t" Nov 03 18:44:14.023201 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.023121 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d8l56"] Nov 03 18:44:14.023291 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.023226 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.026976 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.026960 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ht2s6"] Nov 03 18:44:14.027052 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.026986 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gjwzc"] Nov 03 18:44:14.027116 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.027071 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d8l56" Nov 03 18:44:14.063603 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.063577 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 18:44:14.063712 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.063586 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:14.063712 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.063620 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 18:44:14.063712 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.063638 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:14.063826 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.063760 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:14.063826 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.063791 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 18:44:14.063826 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.063818 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 18:44:14.063997 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.063973 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 18:44:14.063997 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.063987 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 18:44:14.064084 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.064058 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 18:44:14.064151 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.064136 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 18:44:14.064151 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.064145 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 18:44:14.065877 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.065841 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 18:44:14.088701 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.088674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcphm\" (UniqueName: \"kubernetes.io/projected/ff8e353a-e360-459a-89f9-996bc1257069-kube-api-access-jcphm\") pod \"volume-data-source-validator-58c9f5f46d-vrjn4\" (UID: \"ff8e353a-e360-459a-89f9-996bc1257069\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4" Nov 03 18:44:14.088845 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.088709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:14.088845 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.088739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.088845 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.088805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-trusted-ca\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.088984 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.088876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6dkd\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-kube-api-access-k6dkd\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.088984 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.088929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.088984 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.088966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfhlt\" (UniqueName: \"kubernetes.io/projected/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-kube-api-access-gfhlt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:14.089116 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.088993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-default-certificate\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.089116 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlm5f\" (UniqueName: \"kubernetes.io/projected/d8c875c2-ea97-435e-821a-067bc0d5c58e-kube-api-access-tlm5f\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.089116 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:14.089116 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-bound-sa-token\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.089245 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-installation-pull-secrets\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.089245 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.089245 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-service-ca-bundle\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.089405 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-trusted-ca\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.089405 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f0725d29-b4bf-4d33-8089-2c568ba41b0f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:14.089405 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.089515 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd5jj\" (UniqueName: \"kubernetes.io/projected/f0725d29-b4bf-4d33-8089-2c568ba41b0f-kube-api-access-rd5jj\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:14.089515 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:14.089515 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdknd\" (UniqueName: \"kubernetes.io/projected/a6fe8230-18dc-48a4-aa96-d742e707222d-kube-api-access-zdknd\") pod \"kas-connection-checker-685895c7c9-ht2s6\" (UID: \"a6fe8230-18dc-48a4-aa96-d742e707222d\") " pod="kube-system/kas-connection-checker-685895c7c9-ht2s6" Nov 03 18:44:14.089613 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11e435f9-427b-45ce-971e-f9b184d09b07-serving-cert\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:14.089644 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-bound-sa-token\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.089684 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnd84\" (UniqueName: \"kubernetes.io/projected/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-kube-api-access-dnd84\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.089684 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bs8m\" (UniqueName: \"kubernetes.io/projected/664e8917-01ff-4d2c-923d-c7c65fbb5b56-kube-api-access-6bs8m\") pod \"network-check-source-6f78d59f99-mzf7t\" (UID: \"664e8917-01ff-4d2c-923d-c7c65fbb5b56\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t" Nov 03 18:44:14.089746 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089701 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-certificates\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.089746 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089726 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.089837 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:14.089837 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089782 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-image-registry-private-configuration\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.089911 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-image-registry-private-configuration\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.089911 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.089911 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089875 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-installation-pull-secrets\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.089911 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4snqw\" (UniqueName: \"kubernetes.io/projected/eedc3daa-f8aa-4c65-a525-d5072215f3e7-kube-api-access-4snqw\") pod \"kas-connection-checker-685895c7c9-l6xkx\" (UID: \"eedc3daa-f8aa-4c65-a525-d5072215f3e7\") " pod="kube-system/kas-connection-checker-685895c7c9-l6xkx" Nov 03 18:44:14.089911 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh62q\" (UniqueName: \"kubernetes.io/projected/c2709bdc-ce75-411d-9554-39e7c779bf8b-kube-api-access-mh62q\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:14.090071 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-ca-trust-extracted\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.090071 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9nbb\" (UniqueName: \"kubernetes.io/projected/11e435f9-427b-45ce-971e-f9b184d09b07-kube-api-access-v9nbb\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:14.090071 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-tmp\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.090071 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-snapshots\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.090071 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.089986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-serving-cert\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.090071 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3efe5bf0-abca-490f-b431-a2a72f5bbaee-ca-trust-extracted\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.090071 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc1709e-b549-40fc-9471-02c0d106ee9f-config-volume\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.090071 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1368fbe8-4942-4389-8e74-7dacb032b03e-cert\") pod \"ingress-canary-d8l56\" (UID: \"1368fbe8-4942-4389-8e74-7dacb032b03e\") " pod="openshift-ingress-canary/ingress-canary-d8l56" Nov 03 18:44:14.090071 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjk9\" (UniqueName: \"kubernetes.io/projected/1368fbe8-4942-4389-8e74-7dacb032b03e-kube-api-access-drjk9\") pod \"ingress-canary-d8l56\" (UID: \"1368fbe8-4942-4389-8e74-7dacb032b03e\") " pod="openshift-ingress-canary/ingress-canary-d8l56" Nov 03 18:44:14.090071 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090060 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11e435f9-427b-45ce-971e-f9b184d09b07-config\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-serving-cert\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcns\" (UniqueName: \"kubernetes.io/projected/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-kube-api-access-fzcns\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc1709e-b549-40fc-9471-02c0d106ee9f-tmp-dir\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckq4s\" (UniqueName: \"kubernetes.io/projected/cfc1709e-b549-40fc-9471-02c0d106ee9f-kube-api-access-ckq4s\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-config\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-stats-auth\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hd86\" (UniqueName: \"kubernetes.io/projected/48738200-5d6d-4a3e-8798-700496cd1681-kube-api-access-2hd86\") pod \"kas-connection-checker-685895c7c9-gjwzc\" (UID: \"48738200-5d6d-4a3e-8798-700496cd1681\") " pod="kube-system/kas-connection-checker-685895c7c9-gjwzc" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-certificates\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.090419 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.090296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hphfz\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-kube-api-access-hphfz\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.155231 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.155203 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 18:44:14.156725 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.156707 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 18:44:14.160555 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.160540 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 18:44:14.160624 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.160605 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:14.160974 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.160950 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 18:44:14.160974 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.160961 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 18:44:14.161138 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161098 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 18:44:14.161138 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161114 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 18:44:14.161245 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161151 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 18:44:14.161245 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161185 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 18:44:14.161245 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161188 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 18:44:14.161245 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161216 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 18:44:14.161245 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161221 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rw2l5\"" Nov 03 18:44:14.161245 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161237 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 18:44:14.161518 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161252 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 18:44:14.161518 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161261 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 18:44:14.161518 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161251 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-99m4h\"" Nov 03 18:44:14.161518 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161281 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 18:44:14.161518 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161314 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4xp5l\"" Nov 03 18:44:14.161518 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161320 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 18:44:14.161518 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161377 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 18:44:14.161865 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.161848 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l6xkx"] Nov 03 18:44:14.162514 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.162502 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx"] Nov 03 18:44:14.163276 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163212 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 18:44:14.163276 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163214 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-plzjk"] Nov 03 18:44:14.163276 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163246 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:14.163534 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163431 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:14.163534 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163443 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:14.163534 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163451 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:14.163534 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163445 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 18:44:14.163534 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163463 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 18:44:14.163534 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163474 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:14.163534 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163454 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 18:44:14.163534 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163492 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 18:44:14.163534 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163500 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 18:44:14.163534 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163523 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 18:44:14.164070 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163639 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:14.164070 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163716 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 18:44:14.164070 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.163949 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4"] Nov 03 18:44:14.164267 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.164253 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 18:44:14.164653 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.164638 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9q8tt"] Nov 03 18:44:14.165104 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.165087 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hd86\" (UniqueName: \"kubernetes.io/projected/48738200-5d6d-4a3e-8798-700496cd1681-kube-api-access-2hd86\") pod \"kas-connection-checker-685895c7c9-gjwzc\" (UID: \"48738200-5d6d-4a3e-8798-700496cd1681\") " pod="kube-system/kas-connection-checker-685895c7c9-gjwzc" Nov 03 18:44:14.165250 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.165239 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg"] Nov 03 18:44:14.165880 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.165865 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78c7cd5678-n2phb"] Nov 03 18:44:14.166431 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.166418 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8dd798ffc-wswht"] Nov 03 18:44:14.166971 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.166958 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722"] Nov 03 18:44:14.167570 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.167553 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-48wh4"] Nov 03 18:44:14.168382 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.168366 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67986d9d7f-m6wxv"] Nov 03 18:44:14.169251 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.169235 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t"] Nov 03 18:44:14.169807 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.169795 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d8l56"] Nov 03 18:44:14.170321 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.170312 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4"] Nov 03 18:44:14.170875 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.170857 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9pzjb"] Nov 03 18:44:14.190687 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.190669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-certificates\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.190770 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.190693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphfz\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-kube-api-access-hphfz\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.190770 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.190714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcphm\" (UniqueName: \"kubernetes.io/projected/ff8e353a-e360-459a-89f9-996bc1257069-kube-api-access-jcphm\") pod \"volume-data-source-validator-58c9f5f46d-vrjn4\" (UID: \"ff8e353a-e360-459a-89f9-996bc1257069\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4" Nov 03 18:44:14.190770 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.190729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:14.190770 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.190748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.190964 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.190770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-trusted-ca\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.190964 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.190859 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:44:14.190964 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.190886 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:44:14.190964 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.190904 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78c7cd5678-n2phb: secret "image-registry-tls" not found Nov 03 18:44:14.190964 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.190919 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert podName:0d94b4ea-05d0-4bbd-bf1f-3a39db705964 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:14.690898887 +0000 UTC m=+31.761929236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9q8tt" (UID: "0d94b4ea-05d0-4bbd-bf1f-3a39db705964") : secret "networking-console-plugin-cert" not found Nov 03 18:44:14.190964 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.190954 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls podName:3efe5bf0-abca-490f-b431-a2a72f5bbaee nodeName:}" failed. No retries permitted until 2026-11-03 18:44:14.690937315 +0000 UTC m=+31.761967660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls") pod "image-registry-78c7cd5678-n2phb" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee") : secret "image-registry-tls" not found Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.190981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dkd\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-kube-api-access-k6dkd\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhlt\" (UniqueName: \"kubernetes.io/projected/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-kube-api-access-gfhlt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.191110 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.191121 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67986d9d7f-m6wxv: secret "image-registry-tls" not found Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.191151 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls podName:b18d29b5-b9d5-45fd-a470-b0c664a78aeb nodeName:}" failed. No retries permitted until 2026-11-03 18:44:14.691140773 +0000 UTC m=+31.762171119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls") pod "image-registry-67986d9d7f-m6wxv" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb") : secret "image-registry-tls" not found Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-default-certificate\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlm5f\" (UniqueName: \"kubernetes.io/projected/d8c875c2-ea97-435e-821a-067bc0d5c58e-kube-api-access-tlm5f\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-bound-sa-token\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.191321 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-installation-pull-secrets\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.191381 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls podName:c2709bdc-ce75-411d-9554-39e7c779bf8b nodeName:}" failed. No retries permitted until 2026-11-03 18:44:14.691368214 +0000 UTC m=+31.762398560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-txhfg" (UID: "c2709bdc-ce75-411d-9554-39e7c779bf8b") : secret "samples-operator-tls" not found Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-service-ca-bundle\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.191487 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-trusted-ca\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191462 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-certificates\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f0725d29-b4bf-4d33-8089-2c568ba41b0f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd5jj\" (UniqueName: \"kubernetes.io/projected/f0725d29-b4bf-4d33-8089-2c568ba41b0f-kube-api-access-rd5jj\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdknd\" (UniqueName: \"kubernetes.io/projected/a6fe8230-18dc-48a4-aa96-d742e707222d-kube-api-access-zdknd\") pod \"kas-connection-checker-685895c7c9-ht2s6\" (UID: \"a6fe8230-18dc-48a4-aa96-d742e707222d\") " pod="kube-system/kas-connection-checker-685895c7c9-ht2s6" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191608 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11e435f9-427b-45ce-971e-f9b184d09b07-serving-cert\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-bound-sa-token\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191639 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-trusted-ca\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnd84\" (UniqueName: \"kubernetes.io/projected/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-kube-api-access-dnd84\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bs8m\" (UniqueName: \"kubernetes.io/projected/664e8917-01ff-4d2c-923d-c7c65fbb5b56-kube-api-access-6bs8m\") pod \"network-check-source-6f78d59f99-mzf7t\" (UID: \"664e8917-01ff-4d2c-923d-c7c65fbb5b56\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-certificates\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:14.192233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-image-registry-private-configuration\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191848 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-image-registry-private-configuration\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-installation-pull-secrets\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snqw\" (UniqueName: \"kubernetes.io/projected/eedc3daa-f8aa-4c65-a525-d5072215f3e7-kube-api-access-4snqw\") pod \"kas-connection-checker-685895c7c9-l6xkx\" (UID: \"eedc3daa-f8aa-4c65-a525-d5072215f3e7\") " pod="kube-system/kas-connection-checker-685895c7c9-l6xkx" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh62q\" (UniqueName: \"kubernetes.io/projected/c2709bdc-ce75-411d-9554-39e7c779bf8b-kube-api-access-mh62q\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.191997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-ca-trust-extracted\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9nbb\" (UniqueName: \"kubernetes.io/projected/11e435f9-427b-45ce-971e-f9b184d09b07-kube-api-access-v9nbb\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-tmp\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-snapshots\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-serving-cert\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3efe5bf0-abca-490f-b431-a2a72f5bbaee-ca-trust-extracted\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc1709e-b549-40fc-9471-02c0d106ee9f-config-volume\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1368fbe8-4942-4389-8e74-7dacb032b03e-cert\") pod \"ingress-canary-d8l56\" (UID: \"1368fbe8-4942-4389-8e74-7dacb032b03e\") " pod="openshift-ingress-canary/ingress-canary-d8l56" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192217 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjk9\" (UniqueName: \"kubernetes.io/projected/1368fbe8-4942-4389-8e74-7dacb032b03e-kube-api-access-drjk9\") pod \"ingress-canary-d8l56\" (UID: \"1368fbe8-4942-4389-8e74-7dacb032b03e\") " pod="openshift-ingress-canary/ingress-canary-d8l56" Nov 03 18:44:14.192950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11e435f9-427b-45ce-971e-f9b184d09b07-config\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-serving-cert\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192381 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f0725d29-b4bf-4d33-8089-2c568ba41b0f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcns\" (UniqueName: \"kubernetes.io/projected/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-kube-api-access-fzcns\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc1709e-b549-40fc-9471-02c0d106ee9f-tmp-dir\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckq4s\" (UniqueName: \"kubernetes.io/projected/cfc1709e-b549-40fc-9471-02c0d106ee9f-kube-api-access-ckq4s\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-config\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192574 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-stats-auth\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192752 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc1709e-b549-40fc-9471-02c0d106ee9f-tmp-dir\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.192990 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-trusted-ca\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.193103 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.193528 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc1709e-b549-40fc-9471-02c0d106ee9f-config-volume\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.193706 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.193693 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-service-ca-bundle\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.194458 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.194030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-snapshots\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.194458 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.194098 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-tmp\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.194458 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.194165 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-installation-pull-secrets\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.194458 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.194403 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3efe5bf0-abca-490f-b431-a2a72f5bbaee-ca-trust-extracted\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.194458 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.194415 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-ca-trust-extracted\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.194887 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.195035 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.195125 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.195180 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls podName:cfc1709e-b549-40fc-9471-02c0d106ee9f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:14.695165969 +0000 UTC m=+31.766196313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls") pod "dns-default-9pzjb" (UID: "cfc1709e-b549-40fc-9471-02c0d106ee9f") : secret "dns-default-metrics-tls" not found Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.195214 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11e435f9-427b-45ce-971e-f9b184d09b07-serving-cert\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.195360 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.195438 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-stats-auth\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.195524 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.195568 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls podName:f0725d29-b4bf-4d33-8089-2c568ba41b0f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:14.695554814 +0000 UTC m=+31.766585163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4dlvx" (UID: "f0725d29-b4bf-4d33-8089-2c568ba41b0f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.195866 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11e435f9-427b-45ce-971e-f9b184d09b07-config\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.195872 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-certificates\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.195906 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-default-certificate\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.195993 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:14.695974822 +0000 UTC m=+31.767005169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : configmap references non-existent config key: service-ca.crt Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.196261 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:44:14.196489 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.196263 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-config\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.205442 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.196303 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:14.696289615 +0000 UTC m=+31.767319962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : secret "router-metrics-certs-default" not found Nov 03 18:44:14.205442 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.196521 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:14.205442 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.197241 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1368fbe8-4942-4389-8e74-7dacb032b03e-cert\") pod \"ingress-canary-d8l56\" (UID: \"1368fbe8-4942-4389-8e74-7dacb032b03e\") " pod="openshift-ingress-canary/ingress-canary-d8l56" Nov 03 18:44:14.205442 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.197266 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-serving-cert\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.205442 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.197666 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-installation-pull-secrets\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.205442 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.197999 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-serving-cert\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.205442 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.198235 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-image-registry-private-configuration\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.205442 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.199090 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-image-registry-private-configuration\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.224398 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.224379 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gjwzc" Nov 03 18:44:14.250277 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.250247 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcphm\" (UniqueName: \"kubernetes.io/projected/ff8e353a-e360-459a-89f9-996bc1257069-kube-api-access-jcphm\") pod \"volume-data-source-validator-58c9f5f46d-vrjn4\" (UID: \"ff8e353a-e360-459a-89f9-996bc1257069\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4" Nov 03 18:44:14.250394 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.250256 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphfz\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-kube-api-access-hphfz\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.263861 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.263832 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd5jj\" (UniqueName: \"kubernetes.io/projected/f0725d29-b4bf-4d33-8089-2c568ba41b0f-kube-api-access-rd5jj\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:14.264200 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.264176 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnd84\" (UniqueName: \"kubernetes.io/projected/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-kube-api-access-dnd84\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.264415 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.264396 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlm5f\" (UniqueName: \"kubernetes.io/projected/d8c875c2-ea97-435e-821a-067bc0d5c58e-kube-api-access-tlm5f\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.264824 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.264797 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-bound-sa-token\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.265424 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.265396 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bs8m\" (UniqueName: \"kubernetes.io/projected/664e8917-01ff-4d2c-923d-c7c65fbb5b56-kube-api-access-6bs8m\") pod \"network-check-source-6f78d59f99-mzf7t\" (UID: \"664e8917-01ff-4d2c-923d-c7c65fbb5b56\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t" Nov 03 18:44:14.265583 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.265559 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-bound-sa-token\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.265694 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.265673 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snqw\" (UniqueName: \"kubernetes.io/projected/eedc3daa-f8aa-4c65-a525-d5072215f3e7-kube-api-access-4snqw\") pod \"kas-connection-checker-685895c7c9-l6xkx\" (UID: \"eedc3daa-f8aa-4c65-a525-d5072215f3e7\") " pod="kube-system/kas-connection-checker-685895c7c9-l6xkx" Nov 03 18:44:14.266050 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.266028 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9nbb\" (UniqueName: \"kubernetes.io/projected/11e435f9-427b-45ce-971e-f9b184d09b07-kube-api-access-v9nbb\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:14.266883 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.266720 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfhlt\" (UniqueName: \"kubernetes.io/projected/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-kube-api-access-gfhlt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:14.270095 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.267997 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdknd\" (UniqueName: \"kubernetes.io/projected/a6fe8230-18dc-48a4-aa96-d742e707222d-kube-api-access-zdknd\") pod \"kas-connection-checker-685895c7c9-ht2s6\" (UID: \"a6fe8230-18dc-48a4-aa96-d742e707222d\") " pod="kube-system/kas-connection-checker-685895c7c9-ht2s6" Nov 03 18:44:14.270095 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.268311 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6dkd\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-kube-api-access-k6dkd\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.270095 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.268611 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcns\" (UniqueName: \"kubernetes.io/projected/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-kube-api-access-fzcns\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.270095 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.269782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ht2s6" Nov 03 18:44:14.270861 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.270836 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckq4s\" (UniqueName: \"kubernetes.io/projected/cfc1709e-b549-40fc-9471-02c0d106ee9f-kube-api-access-ckq4s\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.270939 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.270910 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjk9\" (UniqueName: \"kubernetes.io/projected/1368fbe8-4942-4389-8e74-7dacb032b03e-kube-api-access-drjk9\") pod \"ingress-canary-d8l56\" (UID: \"1368fbe8-4942-4389-8e74-7dacb032b03e\") " pod="openshift-ingress-canary/ingress-canary-d8l56" Nov 03 18:44:14.271671 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.271653 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh62q\" (UniqueName: \"kubernetes.io/projected/c2709bdc-ce75-411d-9554-39e7c779bf8b-kube-api-access-mh62q\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:14.277601 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.277584 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l6xkx" Nov 03 18:44:14.282968 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.282949 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:14.315586 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.315560 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:14.319707 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.319687 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4" Nov 03 18:44:14.329554 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.329467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:14.342373 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.341190 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:14.351274 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.351209 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t" Nov 03 18:44:14.367717 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.367560 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d8l56" Nov 03 18:44:14.412571 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.412546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:14.412915 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.412898 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:14.413219 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.413199 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:14.467608 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.463899 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 18:44:14.467608 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.463993 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sjcxp\"" Nov 03 18:44:14.467608 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.464046 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bctnd\"" Nov 03 18:44:14.467608 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.464084 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 18:44:14.467608 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.464116 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7r6fd\"" Nov 03 18:44:14.669133 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.669064 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ht2s6"] Nov 03 18:44:14.669942 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.669819 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gjwzc"] Nov 03 18:44:14.670894 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.670651 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l6xkx"] Nov 03 18:44:14.673468 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:14.673435 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fe8230_18dc_48a4_aa96_d742e707222d.slice/crio-d09ccbf110377bf4cecd4005bc7bd4ddede57b8388f5bbcc66579621f788f861 WatchSource:0}: Error finding container d09ccbf110377bf4cecd4005bc7bd4ddede57b8388f5bbcc66579621f788f861: Status 404 returned error can't find the container with id d09ccbf110377bf4cecd4005bc7bd4ddede57b8388f5bbcc66579621f788f861 Nov 03 18:44:14.674067 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:14.674042 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedc3daa_f8aa_4c65_a525_d5072215f3e7.slice/crio-f636a40a9fd1c0fce1ca6282c880b262c4e5439a5e1a14195e24ab05001fd62f WatchSource:0}: Error finding container f636a40a9fd1c0fce1ca6282c880b262c4e5439a5e1a14195e24ab05001fd62f: Status 404 returned error can't find the container with id f636a40a9fd1c0fce1ca6282c880b262c4e5439a5e1a14195e24ab05001fd62f Nov 03 18:44:14.674779 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:14.674669 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48738200_5d6d_4a3e_8798_700496cd1681.slice/crio-7979b0bff7051d3e7de2ea0bf91f28e31c09df5fe190b58b55d66b3d51b9b5dc WatchSource:0}: Error finding container 7979b0bff7051d3e7de2ea0bf91f28e31c09df5fe190b58b55d66b3d51b9b5dc: Status 404 returned error can't find the container with id 7979b0bff7051d3e7de2ea0bf91f28e31c09df5fe190b58b55d66b3d51b9b5dc Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.697463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.697511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.697560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.697588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.697614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.697649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.697697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.697736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.697878 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.697924 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls podName:cfc1709e-b549-40fc-9471-02c0d106ee9f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:15.697909411 +0000 UTC m=+32.768939757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls") pod "dns-default-9pzjb" (UID: "cfc1709e-b549-40fc-9471-02c0d106ee9f") : secret "dns-default-metrics-tls" not found Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.697980 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.698013 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls podName:f0725d29-b4bf-4d33-8089-2c568ba41b0f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:15.698002985 +0000 UTC m=+32.769033334 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4dlvx" (UID: "f0725d29-b4bf-4d33-8089-2c568ba41b0f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.698065 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.698089 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:15.698082688 +0000 UTC m=+32.769113018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : secret "router-metrics-certs-default" not found Nov 03 18:44:14.698356 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.698130 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:44:14.701140 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.698152 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert podName:0d94b4ea-05d0-4bbd-bf1f-3a39db705964 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:15.698146029 +0000 UTC m=+32.769176360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9q8tt" (UID: "0d94b4ea-05d0-4bbd-bf1f-3a39db705964") : secret "networking-console-plugin-cert" not found Nov 03 18:44:14.701140 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.698197 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:44:14.701140 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.698207 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78c7cd5678-n2phb: secret "image-registry-tls" not found Nov 03 18:44:14.701140 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.698235 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls podName:3efe5bf0-abca-490f-b431-a2a72f5bbaee nodeName:}" failed. No retries permitted until 2026-11-03 18:44:15.698225835 +0000 UTC m=+32.769256166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls") pod "image-registry-78c7cd5678-n2phb" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee") : secret "image-registry-tls" not found Nov 03 18:44:14.701140 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.698297 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:44:14.701140 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.698307 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67986d9d7f-m6wxv: secret "image-registry-tls" not found Nov 03 18:44:14.701496 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.701472 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls podName:b18d29b5-b9d5-45fd-a470-b0c664a78aeb nodeName:}" failed. No retries permitted until 2026-11-03 18:44:15.701450837 +0000 UTC m=+32.772481182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls") pod "image-registry-67986d9d7f-m6wxv" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb") : secret "image-registry-tls" not found Nov 03 18:44:14.701593 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.701576 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:44:14.701645 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.701615 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls podName:c2709bdc-ce75-411d-9554-39e7c779bf8b nodeName:}" failed. No retries permitted until 2026-11-03 18:44:15.701604572 +0000 UTC m=+32.772634904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-txhfg" (UID: "c2709bdc-ce75-411d-9554-39e7c779bf8b") : secret "samples-operator-tls" not found Nov 03 18:44:14.701712 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:14.701678 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:15.701670236 +0000 UTC m=+32.772700566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : configmap references non-existent config key: service-ca.crt Nov 03 18:44:14.723278 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.723254 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-plzjk"] Nov 03 18:44:14.728897 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:14.728093 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a9ac1d_c1e8_49d3_9058_72d10b7f662b.slice/crio-8f37de934dda12deb8df293cfa6462d590bdc4fe22c0c67ca39c3efc38745476 WatchSource:0}: Error finding container 8f37de934dda12deb8df293cfa6462d590bdc4fe22c0c67ca39c3efc38745476: Status 404 returned error can't find the container with id 8f37de934dda12deb8df293cfa6462d590bdc4fe22c0c67ca39c3efc38745476 Nov 03 18:44:14.730183 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.730084 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722"] Nov 03 18:44:14.786749 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.786727 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t"] Nov 03 18:44:14.789172 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:14.789150 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664e8917_01ff_4d2c_923d_c7c65fbb5b56.slice/crio-c4253848f40f7e437c5b80082e603675b3ecceedb6fa596de5b64c88bf2d767c WatchSource:0}: Error finding container c4253848f40f7e437c5b80082e603675b3ecceedb6fa596de5b64c88bf2d767c: Status 404 returned error can't find the container with id c4253848f40f7e437c5b80082e603675b3ecceedb6fa596de5b64c88bf2d767c Nov 03 18:44:14.811007 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.810949 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4"] Nov 03 18:44:14.814301 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:14.813660 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a09a125_03b8_4d7a_9c7d_f9e107dd5aa2.slice/crio-57515afc6b1a5fb4cde2944ac0e2bfe93d334c3e26020b89866eae0ca6f9780f WatchSource:0}: Error finding container 57515afc6b1a5fb4cde2944ac0e2bfe93d334c3e26020b89866eae0ca6f9780f: Status 404 returned error can't find the container with id 57515afc6b1a5fb4cde2944ac0e2bfe93d334c3e26020b89866eae0ca6f9780f Nov 03 18:44:14.814301 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.814077 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4"] Nov 03 18:44:14.816584 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:14.816561 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8e353a_e360_459a_89f9_996bc1257069.slice/crio-bdb98a141e27e76b31379692639ff79bdb66de5dd81a4e65e63d6f94c864d53b WatchSource:0}: Error finding container bdb98a141e27e76b31379692639ff79bdb66de5dd81a4e65e63d6f94c864d53b: Status 404 returned error can't find the container with id bdb98a141e27e76b31379692639ff79bdb66de5dd81a4e65e63d6f94c864d53b Nov 03 18:44:14.836689 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.836668 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-48wh4"] Nov 03 18:44:14.838865 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:14.838846 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40ca2ad_bedb_4aa4_b1fd_3ad6da24fcc1.slice/crio-cafebb2b655f5e25b8577b2bd2f8c8d99b349cc9bab576fbac5f04aefb30353f WatchSource:0}: Error finding container cafebb2b655f5e25b8577b2bd2f8c8d99b349cc9bab576fbac5f04aefb30353f: Status 404 returned error can't find the container with id cafebb2b655f5e25b8577b2bd2f8c8d99b349cc9bab576fbac5f04aefb30353f Nov 03 18:44:14.854123 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:14.854103 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d8l56"] Nov 03 18:44:14.856410 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:14.856312 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1368fbe8_4942_4389_8e74_7dacb032b03e.slice/crio-b63e01500df9d86c13bbcdf095287d4d1a740c88e9c2150df01059d92d68f624 WatchSource:0}: Error finding container b63e01500df9d86c13bbcdf095287d4d1a740c88e9c2150df01059d92d68f624: Status 404 returned error can't find the container with id b63e01500df9d86c13bbcdf095287d4d1a740c88e9c2150df01059d92d68f624 Nov 03 18:44:15.630473 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.630440 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4" event={"ID":"ff8e353a-e360-459a-89f9-996bc1257069","Type":"ContainerStarted","Data":"bdb98a141e27e76b31379692639ff79bdb66de5dd81a4e65e63d6f94c864d53b"} Nov 03 18:44:15.631498 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.631474 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" event={"ID":"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2","Type":"ContainerStarted","Data":"57515afc6b1a5fb4cde2944ac0e2bfe93d334c3e26020b89866eae0ca6f9780f"} Nov 03 18:44:15.632520 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.632500 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" event={"ID":"11e435f9-427b-45ce-971e-f9b184d09b07","Type":"ContainerStarted","Data":"d338696153e6a7b4c73119fc8fb55da0029956e7860b4568cbf3312739c84a54"} Nov 03 18:44:15.633843 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.633826 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l6xkx" event={"ID":"eedc3daa-f8aa-4c65-a525-d5072215f3e7","Type":"ContainerStarted","Data":"1610492df872e5f0d4f95957780215f5b2daf4c987398e5d7605aa5de957c33e"} Nov 03 18:44:15.633908 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.633855 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l6xkx" event={"ID":"eedc3daa-f8aa-4c65-a525-d5072215f3e7","Type":"ContainerStarted","Data":"f636a40a9fd1c0fce1ca6282c880b262c4e5439a5e1a14195e24ab05001fd62f"} Nov 03 18:44:15.635143 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.635109 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ht2s6" event={"ID":"a6fe8230-18dc-48a4-aa96-d742e707222d","Type":"ContainerStarted","Data":"be041a0964ad67bb53f8efb3f7ce5d64f529dc23736699485a425271cb76a282"} Nov 03 18:44:15.635229 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.635154 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ht2s6" event={"ID":"a6fe8230-18dc-48a4-aa96-d742e707222d","Type":"ContainerStarted","Data":"d09ccbf110377bf4cecd4005bc7bd4ddede57b8388f5bbcc66579621f788f861"} Nov 03 18:44:15.636233 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.636210 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-plzjk" event={"ID":"17a9ac1d-c1e8-49d3-9058-72d10b7f662b","Type":"ContainerStarted","Data":"8f37de934dda12deb8df293cfa6462d590bdc4fe22c0c67ca39c3efc38745476"} Nov 03 18:44:15.637208 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.637189 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" event={"ID":"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1","Type":"ContainerStarted","Data":"cafebb2b655f5e25b8577b2bd2f8c8d99b349cc9bab576fbac5f04aefb30353f"} Nov 03 18:44:15.638448 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.638430 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gjwzc" event={"ID":"48738200-5d6d-4a3e-8798-700496cd1681","Type":"ContainerStarted","Data":"9470640a90692c45422f255f5d8a2b47d07137ddf14920383c9cf3bb666a2bf0"} Nov 03 18:44:15.638521 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.638466 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gjwzc" event={"ID":"48738200-5d6d-4a3e-8798-700496cd1681","Type":"ContainerStarted","Data":"7979b0bff7051d3e7de2ea0bf91f28e31c09df5fe190b58b55d66b3d51b9b5dc"} Nov 03 18:44:15.639519 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.639500 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d8l56" event={"ID":"1368fbe8-4942-4389-8e74-7dacb032b03e","Type":"ContainerStarted","Data":"b63e01500df9d86c13bbcdf095287d4d1a740c88e9c2150df01059d92d68f624"} Nov 03 18:44:15.640538 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.640517 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t" event={"ID":"664e8917-01ff-4d2c-923d-c7c65fbb5b56","Type":"ContainerStarted","Data":"c4253848f40f7e437c5b80082e603675b3ecceedb6fa596de5b64c88bf2d767c"} Nov 03 18:44:15.707016 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.706987 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:15.707136 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.707040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:15.707136 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.707060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:15.707136 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.707120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:15.707272 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.707148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:15.707272 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707172 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:44:15.707272 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.707175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:15.707272 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707230 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls podName:f0725d29-b4bf-4d33-8089-2c568ba41b0f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:17.707215309 +0000 UTC m=+34.778245646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4dlvx" (UID: "f0725d29-b4bf-4d33-8089-2c568ba41b0f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:44:15.707272 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707264 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:44:15.707493 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.707269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:15.707493 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707267 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:44:15.707493 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707311 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:44:15.707493 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707315 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert podName:0d94b4ea-05d0-4bbd-bf1f-3a39db705964 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:17.707302208 +0000 UTC m=+34.778332537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9q8tt" (UID: "0d94b4ea-05d0-4bbd-bf1f-3a39db705964") : secret "networking-console-plugin-cert" not found Nov 03 18:44:15.707493 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707320 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78c7cd5678-n2phb: secret "image-registry-tls" not found Nov 03 18:44:15.707493 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707340 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:44:15.707493 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707360 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls podName:c2709bdc-ce75-411d-9554-39e7c779bf8b nodeName:}" failed. No retries permitted until 2026-11-03 18:44:17.707346304 +0000 UTC m=+34.778376636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-txhfg" (UID: "c2709bdc-ce75-411d-9554-39e7c779bf8b") : secret "samples-operator-tls" not found Nov 03 18:44:15.707493 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707393 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls podName:3efe5bf0-abca-490f-b431-a2a72f5bbaee nodeName:}" failed. No retries permitted until 2026-11-03 18:44:17.707377381 +0000 UTC m=+34.778407729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls") pod "image-registry-78c7cd5678-n2phb" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee") : secret "image-registry-tls" not found Nov 03 18:44:15.707493 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707415 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:17.70740467 +0000 UTC m=+34.778435027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : secret "router-metrics-certs-default" not found Nov 03 18:44:15.707493 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707470 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:44:15.707493 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707481 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67986d9d7f-m6wxv: secret "image-registry-tls" not found Nov 03 18:44:15.707945 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707514 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls podName:b18d29b5-b9d5-45fd-a470-b0c664a78aeb nodeName:}" failed. No retries permitted until 2026-11-03 18:44:17.707502817 +0000 UTC m=+34.778533165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls") pod "image-registry-67986d9d7f-m6wxv" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb") : secret "image-registry-tls" not found Nov 03 18:44:15.707945 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:15.707476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:15.707945 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707529 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:44:15.707945 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707543 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:17.707532608 +0000 UTC m=+34.778562954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : configmap references non-existent config key: service-ca.crt Nov 03 18:44:15.707945 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:15.707566 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls podName:cfc1709e-b549-40fc-9471-02c0d106ee9f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:17.707553607 +0000 UTC m=+34.778583951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls") pod "dns-default-9pzjb" (UID: "cfc1709e-b549-40fc-9471-02c0d106ee9f") : secret "dns-default-metrics-tls" not found Nov 03 18:44:16.006197 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:16.004856 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-l6xkx" podStartSLOduration=407.004832018 podStartE2EDuration="6m47.004832018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:16.003726066 +0000 UTC m=+33.074756420" watchObservedRunningTime="2026-11-03 18:44:16.004832018 +0000 UTC m=+33.075862372" Nov 03 18:44:16.116592 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:16.115764 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ht2s6" podStartSLOduration=407.115746009 podStartE2EDuration="6m47.115746009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:16.114401617 +0000 UTC m=+33.185431969" watchObservedRunningTime="2026-11-03 18:44:16.115746009 +0000 UTC m=+33.186776362" Nov 03 18:44:16.415004 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:16.414618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:16.415004 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:16.414737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:16.415220 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:16.415161 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 18:44:16.415273 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:16.415243 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs podName:de669e87-278f-48af-bab0-625698973092 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:48.415221812 +0000 UTC m=+65.486252166 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs") pod "network-metrics-daemon-l7j4q" (UID: "de669e87-278f-48af-bab0-625698973092") : secret "metrics-daemon-secret" not found Nov 03 18:44:16.419698 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:16.419673 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:16.516777 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:16.516204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mbx\" (UniqueName: \"kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx\") pod \"network-check-target-g2vhp\" (UID: \"894d2520-9e05-4460-a37d-f3f8377a488b\") " pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:16.521865 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:16.521839 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9mbx\" (UniqueName: \"kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx\") pod \"network-check-target-g2vhp\" (UID: \"894d2520-9e05-4460-a37d-f3f8377a488b\") " pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:16.544425 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:16.544219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:16.585480 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:16.585453 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:16.979492 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:16.978427 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-gjwzc" podStartSLOduration=407.978397688 podStartE2EDuration="6m47.978397688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:16.225648749 +0000 UTC m=+33.296679139" watchObservedRunningTime="2026-11-03 18:44:16.978397688 +0000 UTC m=+34.049428034" Nov 03 18:44:16.982216 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:16.982182 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g2vhp"] Nov 03 18:44:16.983576 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:16.983545 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894d2520_9e05_4460_a37d_f3f8377a488b.slice/crio-d4cdfe4de8fd8d5132b3908a1d3d7a4b4c24f7fcb60dbf27c537acd393760e4f WatchSource:0}: Error finding container d4cdfe4de8fd8d5132b3908a1d3d7a4b4c24f7fcb60dbf27c537acd393760e4f: Status 404 returned error can't find the container with id d4cdfe4de8fd8d5132b3908a1d3d7a4b4c24f7fcb60dbf27c537acd393760e4f Nov 03 18:44:17.008545 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:17.006837 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fn8l9"] Nov 03 18:44:17.009294 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:17.009258 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd42898_f7cc_4198_851b_db51f7ba110a.slice/crio-daa8dea664061fe4640f70d3002e4ceef9c510984edb2adfad6a0a0923cdbafa WatchSource:0}: Error finding container daa8dea664061fe4640f70d3002e4ceef9c510984edb2adfad6a0a0923cdbafa: Status 404 returned error can't find the container with id daa8dea664061fe4640f70d3002e4ceef9c510984edb2adfad6a0a0923cdbafa Nov 03 18:44:17.655439 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:17.655349 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g2vhp" event={"ID":"894d2520-9e05-4460-a37d-f3f8377a488b","Type":"ContainerStarted","Data":"d4cdfe4de8fd8d5132b3908a1d3d7a4b4c24f7fcb60dbf27c537acd393760e4f"} Nov 03 18:44:17.659292 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:17.659261 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fn8l9" event={"ID":"dcd42898-f7cc-4198-851b-db51f7ba110a","Type":"ContainerStarted","Data":"daa8dea664061fe4640f70d3002e4ceef9c510984edb2adfad6a0a0923cdbafa"} Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:17.727392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:17.727449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:17.727491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:17.727513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:17.727533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:17.727561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:17.727603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:17.727651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.727784 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.727844 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls podName:cfc1709e-b549-40fc-9471-02c0d106ee9f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:21.727829061 +0000 UTC m=+38.798859414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls") pod "dns-default-9pzjb" (UID: "cfc1709e-b549-40fc-9471-02c0d106ee9f") : secret "dns-default-metrics-tls" not found Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.727895 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.727922 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls podName:f0725d29-b4bf-4d33-8089-2c568ba41b0f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:21.727911979 +0000 UTC m=+38.798942312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4dlvx" (UID: "f0725d29-b4bf-4d33-8089-2c568ba41b0f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.727964 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.727986 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:21.727979908 +0000 UTC m=+38.799010238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : secret "router-metrics-certs-default" not found Nov 03 18:44:17.728790 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.728042 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:44:17.729666 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.728072 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert podName:0d94b4ea-05d0-4bbd-bf1f-3a39db705964 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:21.728062906 +0000 UTC m=+38.799093242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9q8tt" (UID: "0d94b4ea-05d0-4bbd-bf1f-3a39db705964") : secret "networking-console-plugin-cert" not found Nov 03 18:44:17.729666 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.728129 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:44:17.729666 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.728141 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78c7cd5678-n2phb: secret "image-registry-tls" not found Nov 03 18:44:17.729666 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.728173 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls podName:3efe5bf0-abca-490f-b431-a2a72f5bbaee nodeName:}" failed. No retries permitted until 2026-11-03 18:44:21.72816266 +0000 UTC m=+38.799192992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls") pod "image-registry-78c7cd5678-n2phb" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee") : secret "image-registry-tls" not found Nov 03 18:44:17.729666 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.728233 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:44:17.729666 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.728243 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67986d9d7f-m6wxv: secret "image-registry-tls" not found Nov 03 18:44:17.729666 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.728270 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls podName:b18d29b5-b9d5-45fd-a470-b0c664a78aeb nodeName:}" failed. No retries permitted until 2026-11-03 18:44:21.728261205 +0000 UTC m=+38.799291540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls") pod "image-registry-67986d9d7f-m6wxv" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb") : secret "image-registry-tls" not found Nov 03 18:44:17.729666 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.728345 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:44:17.729666 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.728378 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls podName:c2709bdc-ce75-411d-9554-39e7c779bf8b nodeName:}" failed. No retries permitted until 2026-11-03 18:44:21.728366626 +0000 UTC m=+38.799396957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-txhfg" (UID: "c2709bdc-ce75-411d-9554-39e7c779bf8b") : secret "samples-operator-tls" not found Nov 03 18:44:17.729666 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:17.728453 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:21.728443865 +0000 UTC m=+38.799474201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : configmap references non-existent config key: service-ca.crt Nov 03 18:44:18.667282 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:18.667245 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4" event={"ID":"ff8e353a-e360-459a-89f9-996bc1257069","Type":"ContainerStarted","Data":"5331bf56e78ea5eaf020449aee3dd476489edf2b2e74daee2da71bb1961f0855"} Nov 03 18:44:18.780386 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:18.779223 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4" podStartSLOduration=378.32340048 podStartE2EDuration="6m21.77920852s" totalImagesPullingTime="3.45580804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:18.778543379 +0000 UTC m=+35.849573733" watchObservedRunningTime="2026-11-03 18:44:18.77920852 +0000 UTC m=+35.850238872" Nov 03 18:44:19.195413 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:19.195365 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:19.549958 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:19.549922 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:19.676696 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:19.676676 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/0.log" Nov 03 18:44:19.677026 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:19.676709 2579 generic.go:356] "Generic (PLEG): container finished" podID="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" containerID="231142883c2d91ffcd291aaea7604a6c1d24397c8b10873ecebb91e9f27c371f" exitCode=255 Nov 03 18:44:19.677130 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:19.677111 2579 scope.go:122] "RemoveContainer" containerID="231142883c2d91ffcd291aaea7604a6c1d24397c8b10873ecebb91e9f27c371f" Nov 03 18:44:19.677196 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:19.677130 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" event={"ID":"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1","Type":"ContainerDied","Data":"231142883c2d91ffcd291aaea7604a6c1d24397c8b10873ecebb91e9f27c371f"} Nov 03 18:44:20.217359 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:20.217268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:20.523413 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:20.521075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:20.682732 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:20.682648 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/1.log" Nov 03 18:44:20.683797 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:20.683441 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/0.log" Nov 03 18:44:20.683797 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:20.683475 2579 generic.go:356] "Generic (PLEG): container finished" podID="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" containerID="f7f6eee515b8b8400434a0f9035f753a42c29660c2a778386f698bfae79d48cc" exitCode=255 Nov 03 18:44:20.683797 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:20.683568 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" event={"ID":"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1","Type":"ContainerDied","Data":"f7f6eee515b8b8400434a0f9035f753a42c29660c2a778386f698bfae79d48cc"} Nov 03 18:44:20.683797 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:20.683618 2579 scope.go:122] "RemoveContainer" containerID="231142883c2d91ffcd291aaea7604a6c1d24397c8b10873ecebb91e9f27c371f" Nov 03 18:44:20.684592 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:20.684573 2579 scope.go:122] "RemoveContainer" containerID="f7f6eee515b8b8400434a0f9035f753a42c29660c2a778386f698bfae79d48cc" Nov 03 18:44:20.684859 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:20.684807 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-48wh4_openshift-console-operator(d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" podUID="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" Nov 03 18:44:20.688870 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:20.688843 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" event={"ID":"11e435f9-427b-45ce-971e-f9b184d09b07","Type":"ContainerStarted","Data":"b84f80a963703bc608a6d8725e5221bc670d472e1c8d9cc720bd531f1cb94e43"} Nov 03 18:44:20.691527 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:20.691507 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-plzjk" event={"ID":"17a9ac1d-c1e8-49d3-9058-72d10b7f662b","Type":"ContainerStarted","Data":"3ce8261598a334d86ea973874ac1972db91996eef3e8030900cca6ff1f3bbb8a"} Nov 03 18:44:20.856108 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:20.856033 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:20.913950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:20.913895 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-plzjk" podStartSLOduration=379.030569274 podStartE2EDuration="6m23.913877804s" totalImagesPullingTime="4.88330853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:20.913010361 +0000 UTC m=+37.984040710" watchObservedRunningTime="2026-11-03 18:44:20.913877804 +0000 UTC m=+37.984908157" Nov 03 18:44:21.027199 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.027134 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" podStartSLOduration=378.965813872 podStartE2EDuration="6m24.02711571s" totalImagesPullingTime="5.061301838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:21.025706324 +0000 UTC m=+38.096736677" watchObservedRunningTime="2026-11-03 18:44:21.02711571 +0000 UTC m=+38.098146066" Nov 03 18:44:21.094431 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.094392 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:21.183848 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.183549 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w77sj"] Nov 03 18:44:21.207480 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.206604 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.246193 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.246158 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w77sj"] Nov 03 18:44:21.258857 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.257535 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 18:44:21.258857 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.257649 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gq4wc\"" Nov 03 18:44:21.267900 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.267874 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 18:44:21.369012 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.368975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51924e26-cd42-4c49-9307-f041b7ab2d92-data-volume\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.369171 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.369018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.369171 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.369148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51924e26-cd42-4c49-9307-f041b7ab2d92-crio-socket\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.369262 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.369181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5vg9\" (UniqueName: \"kubernetes.io/projected/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-api-access-r5vg9\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.369262 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.369225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.450950 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.450872 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-7nsd9-jcfjp"] Nov 03 18:44:21.467361 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.464267 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.470197 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.470164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51924e26-cd42-4c49-9307-f041b7ab2d92-data-volume\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.470373 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.470209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.470373 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.470252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51924e26-cd42-4c49-9307-f041b7ab2d92-crio-socket\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.470373 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.470277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5vg9\" (UniqueName: \"kubernetes.io/projected/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-api-access-r5vg9\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.470373 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.470342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.470589 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.470514 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:44:21.470589 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.470568 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls podName:51924e26-cd42-4c49-9307-f041b7ab2d92 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:21.970556904 +0000 UTC m=+39.041587234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w77sj" (UID: "51924e26-cd42-4c49-9307-f041b7ab2d92") : secret "insights-runtime-extractor-tls" not found Nov 03 18:44:21.470974 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.470951 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51924e26-cd42-4c49-9307-f041b7ab2d92-data-volume\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.471090 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.471027 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 18:44:21.471090 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.471072 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm podName:51924e26-cd42-4c49-9307-f041b7ab2d92 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:21.971057616 +0000 UTC m=+39.042087946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w77sj" (UID: "51924e26-cd42-4c49-9307-f041b7ab2d92") : configmap "kube-rbac-proxy" not found Nov 03 18:44:21.471232 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.471127 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51924e26-cd42-4c49-9307-f041b7ab2d92-crio-socket\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.512076 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.512045 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7nsd9-jcfjp"] Nov 03 18:44:21.518287 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.518265 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-l6xrr\"" Nov 03 18:44:21.525731 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.525709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:21.537680 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.537655 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5vg9\" (UniqueName: \"kubernetes.io/projected/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-api-access-r5vg9\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.571075 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.571045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-service-ca-bundle\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.571252 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.571233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgkzv\" (UniqueName: \"kubernetes.io/projected/d03ba26a-59c3-47ce-988a-a4b459714565-kube-api-access-sgkzv\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.571339 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.571272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d03ba26a-59c3-47ce-988a-a4b459714565-archives-path\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.571472 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.571450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-trusted-ca-bundle\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.672303 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.672274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-trusted-ca-bundle\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.672479 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.672361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-service-ca-bundle\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.672541 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.672477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgkzv\" (UniqueName: \"kubernetes.io/projected/d03ba26a-59c3-47ce-988a-a4b459714565-kube-api-access-sgkzv\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.672541 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.672524 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d03ba26a-59c3-47ce-988a-a4b459714565-archives-path\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.673026 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.672954 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d03ba26a-59c3-47ce-988a-a4b459714565-archives-path\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.673026 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.672991 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-service-ca-bundle\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.673212 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.673186 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-trusted-ca-bundle\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.696803 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.696762 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" event={"ID":"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2","Type":"ContainerStarted","Data":"aff2b7547e2e5d179bbfaa5d3f6388af9b1953a0f218acab735d148443186e85"} Nov 03 18:44:21.698471 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.698454 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/1.log" Nov 03 18:44:21.698721 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.698706 2579 scope.go:122] "RemoveContainer" containerID="f7f6eee515b8b8400434a0f9035f753a42c29660c2a778386f698bfae79d48cc" Nov 03 18:44:21.698878 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.698860 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-48wh4_openshift-console-operator(d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" podUID="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" Nov 03 18:44:21.733299 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.733244 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgkzv\" (UniqueName: \"kubernetes.io/projected/d03ba26a-59c3-47ce-988a-a4b459714565-kube-api-access-sgkzv\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.765184 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.765159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:21.773823 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.773796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:21.773941 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.773859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:21.773941 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.773922 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.77390447 +0000 UTC m=+46.844934805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : configmap references non-existent config key: service-ca.crt Nov 03 18:44:21.774058 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.773958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:21.774058 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.774005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:21.774058 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774015 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:44:21.774058 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.774047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:21.774058 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774056 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774070 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls podName:cfc1709e-b549-40fc-9471-02c0d106ee9f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.774054199 +0000 UTC m=+46.845084543 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls") pod "dns-default-9pzjb" (UID: "cfc1709e-b549-40fc-9471-02c0d106ee9f") : secret "dns-default-metrics-tls" not found Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774074 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774105 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls podName:f0725d29-b4bf-4d33-8089-2c568ba41b0f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.774091522 +0000 UTC m=+46.845121869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4dlvx" (UID: "f0725d29-b4bf-4d33-8089-2c568ba41b0f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.774128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774140 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.774156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.774177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774178 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774192 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78c7cd5678-n2phb: secret "image-registry-tls" not found Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774196 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.774183198 +0000 UTC m=+46.845213528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : secret "router-metrics-certs-default" not found Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774228 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert podName:0d94b4ea-05d0-4bbd-bf1f-3a39db705964 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.774218293 +0000 UTC m=+46.845248625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9q8tt" (UID: "0d94b4ea-05d0-4bbd-bf1f-3a39db705964") : secret "networking-console-plugin-cert" not found Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774243 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:44:21.774248 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774248 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls podName:3efe5bf0-abca-490f-b431-a2a72f5bbaee nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.774239062 +0000 UTC m=+46.845269394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls") pod "image-registry-78c7cd5678-n2phb" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee") : secret "image-registry-tls" not found Nov 03 18:44:21.774694 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774275 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls podName:c2709bdc-ce75-411d-9554-39e7c779bf8b nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.774264282 +0000 UTC m=+46.845294616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-txhfg" (UID: "c2709bdc-ce75-411d-9554-39e7c779bf8b") : secret "samples-operator-tls" not found Nov 03 18:44:21.774694 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774279 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:44:21.774694 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774292 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67986d9d7f-m6wxv: secret "image-registry-tls" not found Nov 03 18:44:21.774694 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.774354 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls podName:b18d29b5-b9d5-45fd-a470-b0c664a78aeb nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.774343865 +0000 UTC m=+46.845374209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls") pod "image-registry-67986d9d7f-m6wxv" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb") : secret "image-registry-tls" not found Nov 03 18:44:21.794415 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.794394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:21.809459 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.809420 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" podStartSLOduration=378.738618522 podStartE2EDuration="6m24.809408215s" totalImagesPullingTime="6.070789693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:21.808146094 +0000 UTC m=+38.879176445" watchObservedRunningTime="2026-11-03 18:44:21.809408215 +0000 UTC m=+38.880438566" Nov 03 18:44:21.976774 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.976742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.976929 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:21.976810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:21.976929 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.976862 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 18:44:21.976929 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.976920 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:44:21.976929 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.976930 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm podName:51924e26-cd42-4c49-9307-f041b7ab2d92 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:22.976912853 +0000 UTC m=+40.047943196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w77sj" (UID: "51924e26-cd42-4c49-9307-f041b7ab2d92") : configmap "kube-rbac-proxy" not found Nov 03 18:44:21.977073 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:21.976955 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls podName:51924e26-cd42-4c49-9307-f041b7ab2d92 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:22.976945003 +0000 UTC m=+40.047975351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w77sj" (UID: "51924e26-cd42-4c49-9307-f041b7ab2d92") : secret "insights-runtime-extractor-tls" not found Nov 03 18:44:22.015877 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.015731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:22.370674 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.370650 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7nsd9-jcfjp"] Nov 03 18:44:22.514760 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.514708 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mwz7s"] Nov 03 18:44:22.534381 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.534352 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:22.577578 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.577545 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mwz7s"] Nov 03 18:44:22.585261 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.585232 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 18:44:22.585504 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.585483 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 18:44:22.585504 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.585499 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 18:44:22.585504 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.585488 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 18:44:22.585714 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.585578 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kqw67\"" Nov 03 18:44:22.682997 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.682965 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3a57dde4-90d4-4240-b616-67e07eb614d2-signing-key\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:22.683147 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.683005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3a57dde4-90d4-4240-b616-67e07eb614d2-signing-cabundle\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:22.683147 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.683032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkqk7\" (UniqueName: \"kubernetes.io/projected/3a57dde4-90d4-4240-b616-67e07eb614d2-kube-api-access-vkqk7\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:22.702707 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.702674 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" event={"ID":"d03ba26a-59c3-47ce-988a-a4b459714565","Type":"ContainerStarted","Data":"9f218808178fb1a708d2753dabb989f1656be7c6bd16e9febe0fea4ea6e0b5a7"} Nov 03 18:44:22.703126 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.702731 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" event={"ID":"d03ba26a-59c3-47ce-988a-a4b459714565","Type":"ContainerStarted","Data":"8c67c8d21f94be880836d9c751957db2c005ea0bcb8f22679bc6a1694d7158a7"} Nov 03 18:44:22.704302 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.704278 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d8l56" event={"ID":"1368fbe8-4942-4389-8e74-7dacb032b03e","Type":"ContainerStarted","Data":"347fcccdd3c9f0bccdbdaa9b42a6173d9f3d501cec36e26fc9c13e73dcffe493"} Nov 03 18:44:22.705847 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.705827 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t" event={"ID":"664e8917-01ff-4d2c-923d-c7c65fbb5b56","Type":"ContainerStarted","Data":"ec4660362e6f80ccdee8881f29d0aff7797a1a0b37b5bfbd7a1270a9160576dc"} Nov 03 18:44:22.711232 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.711206 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g2vhp" event={"ID":"894d2520-9e05-4460-a37d-f3f8377a488b","Type":"ContainerStarted","Data":"8531cc799c2a9de7bbb94f63817bc18c11c15ee923ef4b0eaff12fa75e6c0cb4"} Nov 03 18:44:22.711443 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.711427 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:22.785380 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.784809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3a57dde4-90d4-4240-b616-67e07eb614d2-signing-key\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:22.785380 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.784871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3a57dde4-90d4-4240-b616-67e07eb614d2-signing-cabundle\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:22.785380 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.784900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkqk7\" (UniqueName: \"kubernetes.io/projected/3a57dde4-90d4-4240-b616-67e07eb614d2-kube-api-access-vkqk7\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:22.785870 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.785842 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3a57dde4-90d4-4240-b616-67e07eb614d2-signing-cabundle\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:22.787479 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.787457 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3a57dde4-90d4-4240-b616-67e07eb614d2-signing-key\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:22.821378 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.821267 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" podStartSLOduration=1.8212562939999999 podStartE2EDuration="1.821256294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:22.820963023 +0000 UTC m=+39.891993375" watchObservedRunningTime="2026-11-03 18:44:22.821256294 +0000 UTC m=+39.892286645" Nov 03 18:44:22.827719 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.827690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:22.851123 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.851103 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkqk7\" (UniqueName: \"kubernetes.io/projected/3a57dde4-90d4-4240-b616-67e07eb614d2-kube-api-access-vkqk7\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:22.858814 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.858792 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:22.911994 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.911966 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb"] Nov 03 18:44:22.933081 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.932914 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t" podStartSLOduration=376.718880988 podStartE2EDuration="6m23.932899252s" totalImagesPullingTime="7.214018264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:22.931743216 +0000 UTC m=+40.002773561" watchObservedRunningTime="2026-11-03 18:44:22.932899252 +0000 UTC m=+40.003929601" Nov 03 18:44:22.940774 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.940541 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk"] Nov 03 18:44:22.941722 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.941705 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb" Nov 03 18:44:22.962532 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.962504 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk" Nov 03 18:44:22.971611 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.971585 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb"] Nov 03 18:44:22.983098 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.983071 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk"] Nov 03 18:44:22.987428 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.987398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:22.987532 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.987487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:22.987532 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:22.987500 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 18:44:22.987622 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:22.987556 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm podName:51924e26-cd42-4c49-9307-f041b7ab2d92 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:24.987539076 +0000 UTC m=+42.058569406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w77sj" (UID: "51924e26-cd42-4c49-9307-f041b7ab2d92") : configmap "kube-rbac-proxy" not found Nov 03 18:44:22.987679 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:22.987624 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:44:22.987679 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:22.987666 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls podName:51924e26-cd42-4c49-9307-f041b7ab2d92 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:24.987655964 +0000 UTC m=+42.058686294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w77sj" (UID: "51924e26-cd42-4c49-9307-f041b7ab2d92") : secret "insights-runtime-extractor-tls" not found Nov 03 18:44:22.992675 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.992656 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:22.992675 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.992672 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 18:44:22.992789 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:22.992715 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hsjrx\"" Nov 03 18:44:23.042528 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.042479 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g2vhp" podStartSLOduration=33.742559737 podStartE2EDuration="39.042466155s" totalImagesPullingTime="5.299906418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:23.041952919 +0000 UTC m=+40.112983268" watchObservedRunningTime="2026-11-03 18:44:23.042466155 +0000 UTC m=+40.113496510" Nov 03 18:44:23.088048 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.087982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpvtm\" (UniqueName: \"kubernetes.io/projected/ad08468c-b770-47bb-82b9-8c0d4d32f855-kube-api-access-vpvtm\") pod \"migrator-66df5d496b-tqfcb\" (UID: \"ad08468c-b770-47bb-82b9-8c0d4d32f855\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb" Nov 03 18:44:23.088192 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.088050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cxrc\" (UniqueName: \"kubernetes.io/projected/d2ca4100-fa85-49bd-bf74-1339c6116298-kube-api-access-4cxrc\") pod \"migrator-66df5d496b-jnnvk\" (UID: \"d2ca4100-fa85-49bd-bf74-1339c6116298\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk" Nov 03 18:44:23.123425 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.123395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:23.161218 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.161172 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d8l56" podStartSLOduration=3.024199467 podStartE2EDuration="10.161158407s" totalImagesPullingTime="7.13695894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:23.159938778 +0000 UTC m=+40.230969129" watchObservedRunningTime="2026-11-03 18:44:23.161158407 +0000 UTC m=+40.232188760" Nov 03 18:44:23.189323 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.189280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpvtm\" (UniqueName: \"kubernetes.io/projected/ad08468c-b770-47bb-82b9-8c0d4d32f855-kube-api-access-vpvtm\") pod \"migrator-66df5d496b-tqfcb\" (UID: \"ad08468c-b770-47bb-82b9-8c0d4d32f855\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb" Nov 03 18:44:23.189501 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.189418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cxrc\" (UniqueName: \"kubernetes.io/projected/d2ca4100-fa85-49bd-bf74-1339c6116298-kube-api-access-4cxrc\") pod \"migrator-66df5d496b-jnnvk\" (UID: \"d2ca4100-fa85-49bd-bf74-1339c6116298\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk" Nov 03 18:44:23.254948 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.254928 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mwz7s"] Nov 03 18:44:23.255110 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.255091 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cxrc\" (UniqueName: \"kubernetes.io/projected/d2ca4100-fa85-49bd-bf74-1339c6116298-kube-api-access-4cxrc\") pod \"migrator-66df5d496b-jnnvk\" (UID: \"d2ca4100-fa85-49bd-bf74-1339c6116298\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk" Nov 03 18:44:23.255449 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.255434 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpvtm\" (UniqueName: \"kubernetes.io/projected/ad08468c-b770-47bb-82b9-8c0d4d32f855-kube-api-access-vpvtm\") pod \"migrator-66df5d496b-tqfcb\" (UID: \"ad08468c-b770-47bb-82b9-8c0d4d32f855\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb" Nov 03 18:44:23.256693 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:23.256671 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a57dde4_90d4_4240_b616_67e07eb614d2.slice/crio-afaeecf6b369ab7e05ff7a979bf260e93da4c44777f124f49c2eb75881473e75 WatchSource:0}: Error finding container afaeecf6b369ab7e05ff7a979bf260e93da4c44777f124f49c2eb75881473e75: Status 404 returned error can't find the container with id afaeecf6b369ab7e05ff7a979bf260e93da4c44777f124f49c2eb75881473e75 Nov 03 18:44:23.269038 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.268867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb" Nov 03 18:44:23.291212 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.291183 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk" Nov 03 18:44:23.399900 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.399847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:23.664817 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.664796 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb"] Nov 03 18:44:23.666680 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:23.666635 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad08468c_b770_47bb_82b9_8c0d4d32f855.slice/crio-2e57c8db9905549439102b4db00502eb711a1e4082222aefdd606a8f9337360e WatchSource:0}: Error finding container 2e57c8db9905549439102b4db00502eb711a1e4082222aefdd606a8f9337360e: Status 404 returned error can't find the container with id 2e57c8db9905549439102b4db00502eb711a1e4082222aefdd606a8f9337360e Nov 03 18:44:23.712080 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.712052 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk"] Nov 03 18:44:23.713551 ip-10-0-139-179 kubenswrapper[2579]: W1103 18:44:23.713529 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ca4100_fa85_49bd_bf74_1339c6116298.slice/crio-da6a00e7da63135746f3801060339751145d0c899b0a5410f6c6e5197ec569e8 WatchSource:0}: Error finding container da6a00e7da63135746f3801060339751145d0c899b0a5410f6c6e5197ec569e8: Status 404 returned error can't find the container with id da6a00e7da63135746f3801060339751145d0c899b0a5410f6c6e5197ec569e8 Nov 03 18:44:23.718120 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.718096 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" event={"ID":"3a57dde4-90d4-4240-b616-67e07eb614d2","Type":"ContainerStarted","Data":"3086346ad380b7613e7304d9277196bd4a637b0339d837ee6678580701196551"} Nov 03 18:44:23.718206 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.718153 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" event={"ID":"3a57dde4-90d4-4240-b616-67e07eb614d2","Type":"ContainerStarted","Data":"afaeecf6b369ab7e05ff7a979bf260e93da4c44777f124f49c2eb75881473e75"} Nov 03 18:44:23.719382 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.719361 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb" event={"ID":"ad08468c-b770-47bb-82b9-8c0d4d32f855","Type":"ContainerStarted","Data":"2e57c8db9905549439102b4db00502eb711a1e4082222aefdd606a8f9337360e"} Nov 03 18:44:23.831894 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:23.831840 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" podStartSLOduration=1.83182322 podStartE2EDuration="1.83182322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:23.830143805 +0000 UTC m=+40.901174156" watchObservedRunningTime="2026-11-03 18:44:23.83182322 +0000 UTC m=+40.902853572" Nov 03 18:44:24.083746 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:24.083660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:24.330030 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:24.329992 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:24.330204 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:24.330077 2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:24.330515 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:24.330496 2579 scope.go:122] "RemoveContainer" containerID="f7f6eee515b8b8400434a0f9035f753a42c29660c2a778386f698bfae79d48cc" Nov 03 18:44:24.330758 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:24.330730 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-48wh4_openshift-console-operator(d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" podUID="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" Nov 03 18:44:24.727193 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:24.727153 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk" event={"ID":"d2ca4100-fa85-49bd-bf74-1339c6116298","Type":"ContainerStarted","Data":"da6a00e7da63135746f3801060339751145d0c899b0a5410f6c6e5197ec569e8"} Nov 03 18:44:24.747353 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:24.747309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:25.009575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:25.008734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:25.009575 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:25.008822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:25.009575 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:25.009014 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:44:25.009575 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:25.009082 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls podName:51924e26-cd42-4c49-9307-f041b7ab2d92 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.009065697 +0000 UTC m=+46.080096031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w77sj" (UID: "51924e26-cd42-4c49-9307-f041b7ab2d92") : secret "insights-runtime-extractor-tls" not found Nov 03 18:44:25.009575 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:25.009450 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 18:44:25.009575 ip-10-0-139-179 kubenswrapper[2579]: E1103 18:44:25.009494 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm podName:51924e26-cd42-4c49-9307-f041b7ab2d92 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.009480318 +0000 UTC m=+46.080510657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w77sj" (UID: "51924e26-cd42-4c49-9307-f041b7ab2d92") : configmap "kube-rbac-proxy" not found Nov 03 18:44:25.313195 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:25.313113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:25.328430 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:25.328344 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dlbxv_b0169d01-fdae-40a1-ad57-8c380c023d61/dns-node-resolver/0.log" Nov 03 18:44:25.576863 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:25.576790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:25.733073 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:25.733040 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fn8l9" event={"ID":"dcd42898-f7cc-4198-851b-db51f7ba110a","Type":"ContainerStarted","Data":"6a8626ccf4c5c0dce6a43a575ddeb0e289165d20c8b04171e076ee7cb99dedf6"} Nov 03 18:44:25.825228 ip-10-0-139-179 kubenswrapper[2579]: I1103 18:44:25.825196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:25.930660 ip-10-0-139-179 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 18:44:25.946837 ip-10-0-139-179 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 18:44:25.947086 ip-10-0-139-179 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 18:44:25.947153 ip-10-0-139-179 systemd[1]: kubelet.service: Consumed 3.106s CPU time. Nov 03 18:44:25.962921 ip-10-0-139-179 systemd[1]: Starting Kubernetes Kubelet... Nov 03 18:44:26.057434 ip-10-0-139-179 kubenswrapper[5019]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:44:26.057434 ip-10-0-139-179 kubenswrapper[5019]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 18:44:26.057434 ip-10-0-139-179 kubenswrapper[5019]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:44:26.057434 ip-10-0-139-179 kubenswrapper[5019]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:44:26.059828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059743 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:26.059828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059827 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:26.059828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059831 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059835 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059838 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059841 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059843 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059846 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059848 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059851 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059853 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059856 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059858 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059861 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059869 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059872 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059875 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059877 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059880 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059883 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059886 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:26.059967 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059888 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059891 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059894 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059896 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059899 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059902 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059904 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059907 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059909 5019 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059913 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059918 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059921 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059925 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059929 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059931 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059934 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059937 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059940 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059942 5019 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:26.060438 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059945 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059949 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059953 5019 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059955 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059958 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059961 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059963 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059966 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059968 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059972 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059975 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059978 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059980 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059983 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059986 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059989 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059991 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059995 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.059998 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060000 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:26.060909 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060003 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060005 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060008 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060010 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060013 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060017 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060019 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060022 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060024 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060027 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060030 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060032 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060035 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060037 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060040 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060042 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060045 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060047 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060050 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060052 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:26.061415 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060055 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060058 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060060 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060063 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060065 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060068 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060071 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060073 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060076 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060079 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060081 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060084 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060086 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060089 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060091 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060094 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060097 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060100 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060103 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:26.061905 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060106 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060108 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060111 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060113 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060116 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060119 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060122 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060924 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060932 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060935 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060938 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060941 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060944 5019 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060948 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060951 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060953 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060956 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060959 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060961 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060964 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:26.062383 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060966 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060969 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060972 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060974 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060977 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060979 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060982 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060985 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060987 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060990 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060993 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060995 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.060999 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061001 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061004 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061007 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061009 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061012 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061015 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061017 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:26.062864 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061020 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061023 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061025 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061028 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061031 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061033 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061036 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061039 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061041 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061044 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061046 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061049 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061051 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061054 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061056 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061059 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061061 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061064 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061066 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:26.063358 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061069 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061072 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061075 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061077 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061080 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061082 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061092 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061095 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061098 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061101 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061103 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061106 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061109 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061111 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061114 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061123 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061127 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061131 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061133 5019 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061136 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:26.063856 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061139 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061142 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061145 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061147 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061150 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061153 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061155 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061158 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061161 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061164 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061166 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061169 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061172 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061176 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061180 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061183 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061186 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061189 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061191 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:26.064339 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061194 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061197 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061200 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061202 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061205 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061207 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061210 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061213 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061217 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061219 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061222 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061224 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061227 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061229 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061232 5019 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061720 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061725 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061728 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061731 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061734 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:26.064794 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061737 5019 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061739 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061742 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061745 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061747 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061751 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061754 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061757 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061760 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061763 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061766 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061769 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061772 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061775 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061778 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061780 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061783 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061785 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061788 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:26.065303 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061790 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061793 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061796 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061798 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061801 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061804 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061806 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061809 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061812 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061814 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061817 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061820 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061823 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061825 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061828 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061831 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061833 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061836 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061838 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:26.065795 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061841 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061843 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061846 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061849 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061852 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061854 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061857 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061860 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061862 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061865 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061867 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061870 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061872 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061875 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061878 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061880 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061883 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061885 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061888 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061890 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:26.066490 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061893 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061895 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061898 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061900 5019 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061903 5019 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061905 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061908 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061911 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061913 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061916 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061919 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061922 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061924 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061927 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061930 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061932 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061935 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061937 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061940 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061943 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:26.067299 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061946 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061948 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061950 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061953 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061955 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061958 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061961 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061963 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061967 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061971 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061974 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061977 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061980 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061983 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061985 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061988 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061992 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061996 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.061999 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:26.068159 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062002 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062005 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062008 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062010 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062102 5019 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062109 5019 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062115 5019 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062119 5019 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062124 5019 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062127 5019 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062135 5019 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062140 5019 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062143 5019 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062146 5019 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062150 5019 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062153 5019 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062156 5019 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062160 5019 flags.go:64] FLAG: --cgroup-root="" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062163 5019 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062166 5019 flags.go:64] FLAG: --client-ca-file="" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062169 5019 flags.go:64] FLAG: --cloud-provider="external" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062172 5019 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062177 5019 flags.go:64] FLAG: --cluster-domain="" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062180 5019 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 18:44:26.068990 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062183 5019 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062187 5019 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062190 5019 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062194 5019 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062197 5019 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062200 5019 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062203 5019 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062206 5019 flags.go:64] FLAG: --contention-profiling="false" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062209 5019 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062215 5019 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062218 5019 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062221 5019 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062227 5019 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062230 5019 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062233 5019 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062236 5019 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062239 5019 flags.go:64] FLAG: --enable-server="true" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062242 5019 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062246 5019 flags.go:64] FLAG: --event-burst="100" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062249 5019 flags.go:64] FLAG: --event-qps="50" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062253 5019 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062256 5019 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062259 5019 flags.go:64] FLAG: --eviction-hard="" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062263 5019 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 18:44:26.070232 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062266 5019 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062269 5019 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062272 5019 flags.go:64] FLAG: --eviction-soft="" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062275 5019 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062278 5019 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062281 5019 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062284 5019 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062287 5019 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062290 5019 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062293 5019 flags.go:64] FLAG: --feature-gates="" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062297 5019 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062300 5019 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062303 5019 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062306 5019 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062310 5019 flags.go:64] FLAG: --healthz-port="10248" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062313 5019 flags.go:64] FLAG: --help="false" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062316 5019 flags.go:64] FLAG: --hostname-override="ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062321 5019 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062338 5019 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062342 5019 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062347 5019 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062351 5019 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062354 5019 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 18:44:26.071319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062357 5019 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062360 5019 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062363 5019 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062366 5019 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062369 5019 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062372 5019 flags.go:64] FLAG: --kube-reserved="" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062375 5019 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062377 5019 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062380 5019 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062384 5019 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062387 5019 flags.go:64] FLAG: --lock-file="" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062390 5019 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062393 5019 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062396 5019 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062401 5019 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062405 5019 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062408 5019 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062410 5019 flags.go:64] FLAG: --logging-format="text" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062413 5019 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062417 5019 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062420 5019 flags.go:64] FLAG: --manifest-url="" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062423 5019 flags.go:64] FLAG: --manifest-url-header="" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062427 5019 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062431 5019 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062435 5019 flags.go:64] FLAG: --max-pods="110" Nov 03 18:44:26.072315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062438 5019 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062443 5019 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062446 5019 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062448 5019 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062453 5019 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062456 5019 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062459 5019 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062466 5019 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062469 5019 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062472 5019 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062475 5019 flags.go:64] FLAG: --pod-cidr="" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062478 5019 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062481 5019 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062484 5019 flags.go:64] FLAG: --pods-per-core="0" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062487 5019 flags.go:64] FLAG: --port="10250" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062490 5019 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062493 5019 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0fa26096d4518dd05" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062497 5019 flags.go:64] FLAG: --qos-reserved="" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062500 5019 flags.go:64] FLAG: --read-only-port="10255" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062504 5019 flags.go:64] FLAG: --register-node="true" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062507 5019 flags.go:64] FLAG: --register-with-taints="" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062511 5019 flags.go:64] FLAG: --registry-burst="10" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062513 5019 flags.go:64] FLAG: --registry-qps="5" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062516 5019 flags.go:64] FLAG: --reserved-cpus="" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062519 5019 flags.go:64] FLAG: --reserved-memory="" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062523 5019 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 18:44:26.073473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062526 5019 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062529 5019 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062532 5019 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062535 5019 flags.go:64] FLAG: --runonce="false" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062537 5019 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062541 5019 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062544 5019 flags.go:64] FLAG: --seccomp-default="false" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062548 5019 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062552 5019 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062555 5019 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062559 5019 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062562 5019 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062565 5019 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062568 5019 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062571 5019 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062574 5019 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062577 5019 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062580 5019 flags.go:64] FLAG: --system-cgroups="" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062583 5019 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062588 5019 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062591 5019 flags.go:64] FLAG: --tls-cert-file="" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062594 5019 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062599 5019 flags.go:64] FLAG: --tls-min-version="" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062602 5019 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062605 5019 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 18:44:26.074553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062607 5019 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062610 5019 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062613 5019 flags.go:64] FLAG: --v="2" Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062617 5019 flags.go:64] FLAG: --version="false" Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062621 5019 flags.go:64] FLAG: --vmodule="" Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062625 5019 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.062629 5019 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062755 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062758 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062761 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062763 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062766 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062769 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062772 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062776 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062779 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062782 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062786 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062789 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062791 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:26.075647 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062794 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062797 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062800 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062802 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062805 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062808 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062810 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062813 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062816 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062818 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062821 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062824 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062827 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062829 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062832 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062835 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062838 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062840 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062843 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:26.076229 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062845 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062848 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062851 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062853 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062856 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062859 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062861 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062866 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062869 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062871 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062875 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062878 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062881 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062883 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062886 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062889 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062892 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062894 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062897 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062899 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:26.076885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062902 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062904 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062907 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062909 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062912 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062915 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062918 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062920 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062923 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062925 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062928 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062931 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062933 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062936 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062938 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062941 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062944 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062946 5019 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062949 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062953 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:26.077636 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062956 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062959 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062961 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062964 5019 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062967 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062969 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062972 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062975 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062977 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062980 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062982 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062985 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062988 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062990 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062993 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062995 5019 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.062998 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063000 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063003 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063006 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:26.078187 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063009 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063011 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063014 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063017 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063019 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063023 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063027 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063029 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063032 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063035 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063039 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063045 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063049 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063052 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.063077 5019 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:44:26.078854 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063112 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063117 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063120 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063123 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063125 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063128 5019 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063131 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063134 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063136 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063139 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063142 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063145 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063147 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063150 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063152 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063155 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063157 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063161 5019 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063164 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063166 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:26.079241 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063169 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063171 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063174 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063177 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063179 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063182 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063185 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063189 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063192 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063197 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063200 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063203 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063207 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063211 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063214 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063216 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063219 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063221 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063224 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:26.079822 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063227 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063230 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063232 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063235 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063237 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063240 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063243 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063245 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063248 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063251 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063253 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063256 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063259 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063262 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063264 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063267 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063269 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063272 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063274 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:26.080423 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063278 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063281 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063283 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063288 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063290 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063293 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063297 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063299 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063302 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063304 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063307 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063310 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063312 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063315 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063318 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063320 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063336 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063339 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063341 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:26.080882 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063344 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063346 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063349 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063352 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063354 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063357 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063360 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063363 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063366 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063369 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063372 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063374 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063377 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063381 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063384 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063387 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063391 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063394 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063397 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063400 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:26.081341 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063402 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063405 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063407 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063410 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063414 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063416 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063419 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063422 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.063424 5019 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.069103 5019 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.069125 5019 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.069149 5019 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.069157 5019 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069247 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069255 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069262 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069268 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069273 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069277 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069281 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:26.081828 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069301 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069306 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069311 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069315 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069320 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069338 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069343 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069353 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069358 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069363 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069368 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069372 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069377 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069381 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069385 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069389 5019 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069394 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069399 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069404 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069408 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:26.082361 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069412 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069418 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069423 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069427 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069431 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069435 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069440 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069444 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069449 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069453 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069458 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069462 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069467 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069471 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069477 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069484 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069490 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069495 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069499 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069503 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:26.082831 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069509 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069514 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069518 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069524 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069528 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069533 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069537 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069542 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069546 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069551 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069555 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069562 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069568 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069572 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069576 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069581 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069586 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069590 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069595 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:26.083317 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069599 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069604 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069608 5019 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069612 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069617 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069621 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069625 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069630 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069634 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069639 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069643 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069647 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069652 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069668 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069673 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069678 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069682 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069687 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069691 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069695 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:26.083804 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069699 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069704 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069708 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069713 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069717 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069721 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069726 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069730 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069735 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069739 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069743 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069748 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069752 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069757 5019 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069761 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069766 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069770 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069774 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069778 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:26.084349 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.069782 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.069824 5019 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070049 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070056 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070061 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070065 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070070 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070076 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070080 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070084 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070089 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070093 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070097 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070102 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070106 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:26.084958 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070110 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070114 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070118 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070122 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070126 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070130 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070135 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070139 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070144 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070148 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070152 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070158 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070165 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070170 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070175 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070180 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070185 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070190 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070194 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070198 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:26.085481 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070202 5019 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070207 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070211 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070216 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070220 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070226 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070231 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070235 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070239 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070242 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070247 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070251 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070255 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070259 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070262 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070267 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070271 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070274 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070278 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070283 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:26.086385 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070287 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070291 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070296 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070300 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070305 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070309 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070313 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070317 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070321 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070342 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070347 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070351 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070355 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070361 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070365 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070370 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070374 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070379 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070384 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:26.087188 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070388 5019 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070393 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070397 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070402 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070406 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070411 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070415 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070419 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070423 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070427 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070431 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070436 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070440 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070444 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070448 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070453 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070457 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070462 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070466 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070470 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:26.087749 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070474 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070478 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070482 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070487 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070491 5019 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070495 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070499 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070504 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070508 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070513 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070517 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070523 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070532 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070537 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.070584 5019 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070637 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:44:26.088267 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070644 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070650 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070654 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070659 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070664 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070669 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070674 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070679 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070683 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070687 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070692 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070697 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070701 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070705 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070710 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070714 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070718 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070723 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070727 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070731 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:44:26.088709 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070735 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070740 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070744 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070748 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070753 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070757 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070761 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070765 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070770 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070775 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070779 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070784 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070788 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070792 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070797 5019 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070801 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070806 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070810 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070814 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:44:26.089178 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070818 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070822 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070827 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070831 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070835 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070839 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070843 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070847 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070852 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070859 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070863 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070868 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070874 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070878 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070882 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070886 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070891 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070897 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070901 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070905 5019 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:44:26.089690 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070910 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070914 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070918 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070924 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070929 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070933 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070939 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070945 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070949 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070954 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070958 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070963 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070967 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070972 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070976 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070980 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070984 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070988 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070993 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:44:26.090169 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.070997 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071001 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071007 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071012 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071016 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071021 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071025 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071029 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071033 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071037 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071044 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071048 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071052 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071057 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071063 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071069 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071074 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071080 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071084 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071089 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:44:26.090641 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071093 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071097 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071101 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071106 5019 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071110 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071115 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:26.071120 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.071199 5019 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.072710 5019 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.072779 5019 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.073038 5019 server.go:1014] "Starting client certificate rotation" Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.073172 5019 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.073230 5019 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 18:38:43 +0000 UTC" deadline="2028-04-14 02:00:22.021470742 +0000 UTC" Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.073279 5019 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12655h15m55.948195287s" Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.073773 5019 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.075459 5019 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:44:26.091124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.077026 5019 log.go:25] "Validated CRI v1 runtime API" Nov 03 18:44:26.091538 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.078693 5019 log.go:25] "Validated CRI v1 image API" Nov 03 18:44:26.091538 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.079525 5019 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 18:44:26.091538 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.081625 5019 fs.go:135] Filesystem UUIDs: map[72112ea2-b8bf-43ab-95b4-895866284f31:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e70b221b-ded6-42c1-8a22-aa22a3c39021:/dev/nvme0n1p4] Nov 03 18:44:26.091639 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.081644 5019 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13b91cb8dffc8ff97a59294067359465bdeea6fcc392e9e5c8a9542f7ae216f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13b91cb8dffc8ff97a59294067359465bdeea6fcc392e9e5c8a9542f7ae216f4/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ad4518d7c2651fa29049a332906a048239f09fceb8a9dc1a41b5ca1e623ca97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ad4518d7c2651fa29049a332906a048239f09fceb8a9dc1a41b5ca1e623ca97/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e57c8db9905549439102b4db00502eb711a1e4082222aefdd606a8f9337360e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e57c8db9905549439102b4db00502eb711a1e4082222aefdd606a8f9337360e/userdata/shm major:0 minor:361 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33f0f81883389e4be510096c5e425b5cafaa34c1a94a88f4b72aa7f2043b18d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33f0f81883389e4be510096c5e425b5cafaa34c1a94a88f4b72aa7f2043b18d5/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dfa67acd059b2489cdc964514150aefa2d7bb74d2c5599b662be0502804fb85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dfa67acd059b2489cdc964514150aefa2d7bb74d2c5599b662be0502804fb85/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/512f5c3d2b95a5f3a86a58b3129b890070fb21e8b90bd51f1825693e089205de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/512f5c3d2b95a5f3a86a58b3129b890070fb21e8b90bd51f1825693e089205de/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57515afc6b1a5fb4cde2944ac0e2bfe93d334c3e26020b89866eae0ca6f9780f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57515afc6b1a5fb4cde2944ac0e2bfe93d334c3e26020b89866eae0ca6f9780f/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f2127f149bf105f7e6d273f1d355b361c93bebb08b3d440900c100e29b5ce20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f2127f149bf105f7e6d273f1d355b361c93bebb08b3d440900c100e29b5ce20/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7979b0bff7051d3e7de2ea0bf91f28e31c09df5fe190b58b55d66b3d51b9b5dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7979b0bff7051d3e7de2ea0bf91f28e31c09df5fe190b58b55d66b3d51b9b5dc/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c67c8d21f94be880836d9c751957db2c005ea0bcb8f22679bc6a1694d7158a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c67c8d21f94be880836d9c751957db2c005ea0bcb8f22679bc6a1694d7158a7/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f37de934dda12deb8df293cfa6462d590bdc4fe22c0c67ca39c3efc38745476/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f37de934dda12deb8df293cfa6462d590bdc4fe22c0c67ca39c3efc38745476/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9160a39d465ef69ed863605fb4c1f3319a5d102edab38c03ae17f3fba0ebf194/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9160a39d465ef69ed863605fb4c1f3319a5d102edab38c03ae17f3fba0ebf194/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afaeecf6b369ab7e05ff7a979bf260e93da4c44777f124f49c2eb75881473e75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afaeecf6b369ab7e05ff7a979bf260e93da4c44777f124f49c2eb75881473e75/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b63e01500df9d86c13bbcdf095287d4d1a740c88e9c2150df01059d92d68f624/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b63e01500df9d86c13bbcdf095287d4d1a740c88e9c2150df01059d92d68f624/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bdb98a141e27e76b31379692639ff79bdb66de5dd81a4e65e63d6f94c864d53b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bdb98a141e27e76b31379692639ff79bdb66de5dd81a4e65e63d6f94c864d53b/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4253848f40f7e437c5b80082e603675b3ecceedb6fa596de5b64c88bf2d767c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4253848f40f7e437c5b80082e603675b3ecceedb6fa596de5b64c88bf2d767c/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5688730c6f24ee8bcc7d9d40f40fc26f099049f14af184f4c1a9add2b5ebcf1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5688730c6f24ee8bcc7d9d40f40fc26f099049f14af184f4c1a9add2b5ebcf1/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cafebb2b655f5e25b8577b2bd2f8c8d99b349cc9bab576fbac5f04aefb30353f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cafebb2b655f5e25b8577b2bd2f8c8d99b349cc9bab576fbac5f04aefb30353f/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc57b913b081983f0738901b4ba4e13009779804fb419fbe114fb30ed7acbe3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc57b913b081983f0738901b4ba4e13009779804fb419fbe114fb30ed7acbe3c/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d09ccbf110377bf4cecd4005bc7bd4ddede57b8388f5bbcc66579621f788f861/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d09ccbf110377bf4cecd4005bc7bd4ddede57b8388f5bbcc66579621f788f861/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d338696153e6a7b4c73119fc8fb55da0029956e7860b4568cbf3312739c84a54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d338696153e6a7b4c73119fc8fb55da0029956e7860b4568cbf3312739c84a54/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4cdfe4de8fd8d5132b3908a1d3d7a4b4c24f7fcb60dbf27c537acd393760e4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4cdfe4de8fd8d5132b3908a1d3d7a4b4c24f7fcb60dbf27c537acd393760e4f/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da6a00e7da63135746f3801060339751145d0c899b0a5410f6c6e5197ec569e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da6a00e7da63135746f3801060339751145d0c899b0a5410f6c6e5197ec569e8/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/daa8dea664061fe4640f70d3002e4ceef9c510984edb2adfad6a0a0923cdbafa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/daa8dea664061fe4640f70d3002e4ceef9c510984edb2adfad6a0a0923cdbafa/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0205d6051b2957d46025f61c2823434f601e946565b2248eb4caf1af9641af5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0205d6051b2957d46025f61c2823434f601e946565b2248eb4caf1af9641af5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f636a40a9fd1c0fce1ca6282c880b262c4e5439a5e1a14195e24ab05001fd62f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f636a40a9fd1c0fce1ca6282c880b262c4e5439a5e1a14195e24ab05001fd62f/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0f1aae-624b-4735-ae9b-d27b716a6d9c/volumes/kubernetes.io~projected/kube-api-access-bqpjs:{mountpoint:/var/lib/kubelet/pods/0e0f1aae-624b-4735-ae9b-d27b716a6d9c/volumes/kubernetes.io~projected/kube-api-access-bqpjs major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11e435f9-427b-45ce-971e-f9b184d09b07/volumes/kubernetes.io~projected/kube-api-access-v9nbb:{mountpoint:/var/lib/kubelet/pods/11e435f9-427b-45ce-971e-f9b184d09b07/volumes/kubernetes.io~projected/kube-api-access-v9nbb major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11e435f9-427b-45ce-971e-f9b184d09b07/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/11e435f9-427b-45ce-971e-f9b184d09b07/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1368fbe8-4942-4389-8e74-7dacb032b03e/volumes/kubernetes.io~projected/kube-api-access-drjk9:{mountpoint:/var/lib/kubelet/pods/1368fbe8-4942-4389-8e74-7dacb032b03e/volumes/kubernetes.io~projected/kube-api-access-drjk9 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1368fbe8-4942-4389-8e74-7dacb032b03e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1368fbe8-4942-4389-8e74-7dacb032b03e/volumes/kubernetes.io~secret/cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16e9a2c3-c4e3-4e78-9157-926fae93f8a8/volumes/kubernetes.io~projected/kube-api-access-mv7j9:{mountpoint:/var/lib/kubelet/pods/16e9a2c3-c4e3-4e78-9157-926fae93f8a8/volumes/kubernetes.io~projected/kube-api-access-mv7j9 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17a9ac1d-c1e8-49d3-9058-72d10b7f662b/volumes/kubernetes.io~projected/kube-api-access-dnd84:{mountpoint:/var/lib/kubelet/pods/17a9ac1d-c1e8-49d3-9058-72d10b7f662b/volumes/kubernetes.io~projected/kube-api-access-dnd84 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17a9ac1d-c1e8-49d3-9058-72d10b7f662b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/17a9ac1d-c1e8-49d3-9058-72d10b7f662b/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a57dde4-90d4-4240-b616-67e07eb614d2/volumes/kubernetes.io~projected/kube-api-access-vkqk7:{mountpoint:/var/lib/kubelet/pods/3a57dde4-90d4-4240-b616-67e07eb614d2/volumes/kubernetes.io~projected/kube-api-access-vkqk7 major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a57dde4-90d4-4240-b616-67e07eb614d2/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/3a57dde4-90d4-4240-b616-67e07eb614d2/volumes/kubernetes.io~secret/signing-key major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes/kubernetes.io~projected/kube-api-access-k6dkd:{mountpoint:/var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes/kubernetes.io~projected/kube-api-access-k6dkd major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48738200-5d6d-4a3e-8798-700496cd1681/volumes/kubernetes.io~projected/kube-api-access-2hd86:{mountpoint:/var/lib/kubelet/pods/48738200-5d6d-4a3e-8798-700496cd1681/volumes/kubernetes.io~projected/kube-api-access-2hd86 major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51924e26-cd42-4c49-9307-f041b7ab2d92/volumes/kubernetes.io~projected/kube-api-access-r5vg9:{mountpoint:/var/lib/kubelet/pods/51924e26-cd42-4c49-9307-f041b7ab2d92/volumes/kubernetes.io~projected/kube-api-access-r5vg9 major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2/volumes/kubernetes.io~projected/kube-api-access-gfhlt:{mountpoint:/var/lib/kubelet/pods/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2/volumes/kubernetes.io~projected/kube-api-access-gfhlt major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/664e8917-01ff-4d2c-923d-c7c65fbb5b56/volumes/kubernetes.io~projected/kube-api-access-6bs8m:{mountpoint:/var/lib/kubelet/pods/664e8917-01ff-4d2c-923d-c7c65fbb5b56/volumes/kubernetes.io~projected/kube-api-access-6bs8m major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/894d2520-9e05-4460-a37d-f3f8377a488b/volumes/kubernetes.io~projected/kube-api-access-g9mbx:{mountpoint:/var/lib/kubelet/pods/894d2520-9e05-4460-a37d-f3f8377a488b/volumes/kubernetes.io~projected/kube-api-access-g9mbx major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9126b699-4c1b-46ca-976c-a4e1813357a1/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9126b699-4c1b-46ca-976c-a4e1813357a1/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a63bf8a-50a3-4068-9835-50cfd2156350/volumes/kubernetes.io~projected/kube-api-access-vcb52:{mountpoint:/var/lib/kubelet/pods/9a63bf8a-50a3-4068-9835-50cfd2156350/volumes/kubernetes.io~projected/kube-api-access-vcb52 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684/volumes/kubernetes.io~projected/kube-api-access-fhtzm:{mountpoint:/var/lib/kubelet/pods/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684/volumes/kubernetes.io~projected/kube-api-access-fhtzm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6fe8230-18dc-48a4-aa96-d742e707222d/volumes/kubernetes.io~projected/kube-api-access-zdknd:{mountpoint:/var/lib/kubelet/pods/a6fe8230-18dc-48a4-aa96-d742e707222d/volumes/kubernetes.io~projected/kube-api-access-zdknd major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad08468c-b770-47bb-82b9-8c0d4d32f855/volumes/kubernetes.io~projected/kube-api-access-vpvtm:{mountpoint:/var/lib/kubelet/pods/ad08468c-b770-47bb-82b9-8c0d4d32f855/volumes/kubernetes.io~projected/kube-api-access-vpvtm major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0169d01-fdae-40a1-ad57-8c380c023d61/volumes/kubernetes.io~projected/kube-api-access-zzh4l:{mountpoint:/var/lib/kubelet/pods/b0169d01-fdae-40a1-ad57-8c380c023d61/volumes/kubernetes.io~projected/kube-api-access-zzh4l major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes/kubernetes.io~projected/kube-api-access-hphfz:{mountpoint:/var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes/kubernetes.io~projected/kube-api-access-hphfz major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2709bdc-ce75-411d-9554-39e7c779bf8b/volumes/kubernetes.io~projected/kube-api-access-mh62q:{mountpoint:/var/lib/kubelet/pods/c2709bdc-ce75-411d-9554-39e7c779bf8b/volumes/kubernetes.io~projected/kube-api-access-mh62q major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7bddb76-6f22-430e-9016-ce19e959dd48/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c7bddb76-6f22-430e-9016-ce19e959dd48/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7bddb76-6f22-430e-9016-ce19e959dd48/volumes/kubernetes.io~projected/kube-api-access-gl646:{mountpoint:/var/lib/kubelet/pods/c7bddb76-6f22-430e-9016-ce19e959dd48/volumes/kubernetes.io~projected/kube-api-access-gl646 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7bddb76-6f22-430e-9016-ce19e959dd48/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c7bddb76-6f22-430e-9016-ce19e959dd48/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfc1709e-b549-40fc-9471-02c0d106ee9f/volumes/kubernetes.io~projected/kube-api-access-ckq4s:{mountpoint:/var/lib/kubelet/pods/cfc1709e-b549-40fc-9471-02c0d106ee9f/volumes/kubernetes.io~projected/kube-api-access-ckq4s major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d03ba26a-59c3-47ce-988a-a4b459714565/volumes/kubernetes.io~projected/kube-api-access-sgkzv:{mountpoint:/var/lib/kubelet/pods/d03ba26a-59c3-47ce-988a-a4b459714565/volumes/kubernetes.io~projected/kube-api-access-sgkzv major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2ca4100-fa85-49bd-bf74-1339c6116298/volumes/kubernetes.io~projected/kube-api-access-4cxrc:{mountpoint:/var/lib/kubelet/pods/d2ca4100-fa85-49bd-bf74-1339c6116298/volumes/kubernetes.io~projected/kube-api-access-4cxrc major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/volumes/kubernetes.io~projected/kube-api-access-fzcns:{mountpoint:/var/lib/kubelet/pods/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/volumes/kubernetes.io~projected/kube-api-access-fzcns major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d52d4fd1-25a8-44d6-94d8-13757fe7790e/volumes/kubernetes.io~projected/kube-api-access-l8m9t:{mountpoint:/var/lib/kubelet/pods/d52d4fd1-25a8-44d6-94d8-13757fe7790e/volumes/kubernetes.io~projected/kube-api-access-l8m9t major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8c875c2-ea97-435e-821a-067bc0d5c58e/volumes/kubernetes.io~projected/kube-api-access-tlm5f:{mountpoint:/var/lib/kubelet/pods/d8c875c2-ea97-435e-821a-067bc0d5c58e/volumes/kubernetes.io~projected/kube-api-access-tlm5f major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8c875c2-ea97-435e-821a-067bc0d5c58e/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/d8c875c2-ea97-435e-821a-067bc0d5c58e/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8c875c2-ea97-435e-821a-067bc0d5c58e/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/d8c875c2-ea97-435e-821a-067bc0d5c58e/volumes/kubernetes.io~secret/stats-auth major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcd42898-f7cc-4198-851b-db51f7ba110a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/dcd42898-f7cc-4198-851b-db51f7ba110a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de669e87-278f-48af-bab0-625698973092/volumes/kubernetes.io~projected/kube-api-access-zvp79:{mountpoint:/var/lib/kubelet/pods/de669e87-278f-48af-bab0-625698973092/volumes/kubernetes.io~projected/kube-api-access-zvp79 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1055486-a6b2-425f-9c20-89c93df7e080/volumes/kubernetes.io~projected/kube-api-access-z575t:{mountpoint:/var/lib/kubelet/pods/e1055486-a6b2-425f-9c20-89c93df7e080/volumes/kubernetes.io~projected/kube-api-access-z575t major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eedc3daa-f8aa-4c65-a525-d5072215f3e7/volumes/kubernetes.io~projected/kube-api-access-4snqw:{mountpoint:/var/lib/kubelet/pods/eedc3daa-f8aa-4c65-a525-d5072215f3e7/volumes/kubernetes.io~projected/kube-api-access-4snqw major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0725d29-b4bf-4d33-8089-2c568ba41b0f/volumes/kubernetes.io~projected/kube-api-access-rd5jj:{mountpoint:/var/lib/kubelet/pods/f0725d29-b4bf-4d33-8089-2c568ba41b0f/volumes/kubernetes.io~projected/kube-api-access-rd5jj major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff8e353a-e360-459a-89f9-996bc1257069/volumes/kubernetes.io~projected/kube-api-access-jcphm:{mountpoint:/var/lib/kubelet/pods/ff8e353a-e360-459a-89f9-996bc1257069/volumes/kubernetes.io~projected/kube-api-access-jcphm major:0 minor:211 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a9eb6e59d355a411fee9f25f5a3358954819d5fb11423fe86669039327273ede/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/102ab2ef36de65a116b1a0a43008c1ed6aa1dd44b7bdaf3f9e35c14602391074/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e65ad11f5e72bcb5401e55f1c6d00f796780702025c44cc5fe54cf556d46d215/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/ffa8ab26f4ec97a1bee6b6189adcdea69b80cd51e93eac6d232d0f3ec441d6e1/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/116d90e224cc1928c912efa047058261bb7937a3447ecbb1954ca6a17b54791d/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5c57be2dbd98e0df755b49b681fa7605ce8289c11e4e780498a4d6718632fc9e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/507288b41baa95216a9f8b4ea07bdb797d80fbc70cafbacf74efbd96a0c22689/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b3d6de74e6c9faa76e7cd213ff1055c3d67158f3ec7267bc610e99db06e2cc69/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/f68be37335bd183446aca8aa35878a102a6aee91162f33c7946d67a5ce056a13/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/5052eba55ffd85fed61b659a1075a8cc387419756d2ba7cf41ab55b600561ea0/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/9e10ed924a9c3175f045a6e732358b96555ebb26fc9017d4671ab3d76c4fd737/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/c4e6340fd704d571b6fef2545d334e7db5825fa4b1e93f02c1959030b422248c/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a8156fb7fb2ac03ef956d17d577246459cdc7e3c9b0d9e2f1a2362161ce6dadf/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0b51ad8607a4f5b2e9929c1b350edec9d0919f705cf8162ff666ced9eb667bcc/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/522a32d39418909dd86731b728d735296ee3c8c20461f1cf7c1c4597657decc3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7fa3b74869b731c4207cb851fd5b887f1ec8498a8021872f5e02e43848723c0d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/891f020c4efb81001c612536a2b566c6fe0771b8eaf198c762e1f4a8d286e32c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/63caabf9e2f241f9d2460cc4ccbc317b88f55df33713d3b6246aab365759039e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/bd56e0b27c00c68145532bc17fe37509970ee1971720b49d2283802db24d5752/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c8f7f843eba6c5121392bd511526d83664ba8f5b655dd1b5f37af801654b0f10/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5a5e3a09973f7fda79ad6c10b54807d7086ef13ca29ee6730929c6459f4bcb10/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/09efe402c8a32d20e853d3d077c2e158edc8d6fe8b76927fc5dc55a82bb2090b/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/c0a817976e142920726393bc4304bfe82b04895cf6a986bf17d8a44f60048935/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/734a62ef1c02395e2962c79b7034b70d62ef0c932b89ec1358d36a5e381e807a/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/88e16c9c44b9eade77c4e3141228b974758561914fadaee69d180ab015c57ad5/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/5741e81f8d0bc431521c1bda2d597bf7b68b425520b0bfd28446a2fc2cea3524/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/03b65e6fc7e96ce5a141de015e0e12d5a00fc64df0a3d32004f4734e43279910/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/052ac4eace62f82835bcdfcbe280404823017b39c4c1c2e93267e587cba1e363/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/870da76dd81bf0744e783cc7b1d58629db6d8e70d8328cc6b00ac750c5e99c02/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/3968eb0b0d93984adfffc4a0032d8d94f6be3a3d5398c1701217bb5cac3b9d08/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/df7e9c86cc70049fd5fa7dcd8e997f8b8c2ae7dfb6a8d7e8fe3fbc4a302f6884/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/359a02b21f3d09fb1c31df34fe612ff9ea34519a9cbd684f76b863907c1ac85c/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/1d07619ba6116f9b5c84e7dc597b3933ad0620b0acca9c1e7fef4380fe2fef3b/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/694fee7a1f70e4c650e5a23f26ab1ab23d53cc433ac2080cec62a7261aad7168/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/91da5335c26fa81b7ca30cd57443676e3a9b45e8a643addf6b5d4bd0bb5afd2a/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/e4035e168637ca963ecb33f72066c1e662baf533dadaca42b72f0baf2493b841/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/043a53219092db0612dc8f224500afaa3ff60c65cc09711c4d47ec209964fd31/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/a600f7ad71cd9d9cd280c0d088de2b20296db0667a2de7019af356dbfce5728a/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/94f429ba03246046a6b359d69b32a55194ad0575a94a24b80447ef47be99829f/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/01a7cb4e399cf7bfbce869681f29804bebf2aaf20cc883d9be7b8fb002e08305/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/90d483e3619b1d4176e0bb23443b542f05c8d10276a02d7f85749eb172ad2949/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/dc5a6b2347d7bec4694d84610e6f189738d6064e7e31299209e9d1a096ec4440/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/2a39559465c4bec8c59a15ede6667e7be2f402546b417601b95f3cfe72aa6a2e/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/a48648331a1e962f345bf4f54d7d59318c3c03db42bbc614fbe8283d7b6b58d8/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/57787ec1b7b7ee9898e607014c0845728483acac0c4f5bc4f77b5bcbb1d5e740/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/d51b4023c61d4b1264794e36d8d199def2dbb0795faa908959037799aa903955/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/92131bb314ddef3c7b76cb0d0e352ab272f547ec096641de621bed9611ebf0bd/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/43ea611a1c5cf202137860ac3f1d0bb59699661376a9a32c6c4ad32bf22b78cb/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/61e17c926f8e99d13eb7a857aa9ef818c274378950030870bec774c87aa5c090/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/32a4e7e326eed2eca86bc826ea0d28440a42e7b125f2cfc3b7e2dd0dab9244f5/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/51515afad0fe8fdf73cf6cb0447d785a0c93cc0e253a952b33d2cf9c6c5086f9/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/44db135c60a513668a8f01aa1fba0866d19f1537e2f2748cc9d8ce8365257f20/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/e4d31b26f707ed091deb0af11694a8d409dce0b85693e5a0de752b80713b6ded/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/69879d85db84c03974f8e2dc119eb326ed3cbb6994984a8094bc5005d6362152/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bfaa866a68576953d83c372328885f74071223f484568c74eb050e8de13936f3/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/03fbd002befdd42203ed26311a63378f881f7058ee91d1e6a57f19074c3f3f03/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/687f9f360d4f86c4e33aef1521962bae00e63cb89c9745f3fa0805d301064f33/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8d19fdb7cfe5aa2f89687d74fe74445c260e9fa69499df0bb327e6d8fdc3cbc6/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/07ac7725c0a5a3dc0971aec79514412ee9b492b9c03c0f504181dfbf31a31650/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6a1f3cb42af910d58dae6d7b64799a39b77991b2ff734cb8f5d81eb6e9025ba0/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 18:44:26.091848 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.091462 5019 manager.go:217] Machine: {Timestamp:2026-11-03 18:44:26.090985408 +0000 UTC m=+0.063546581 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122824 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec227d25bf2188b50acebe17e5b51ab7 SystemUUID:ec227d25-bf21-88b5-0ace-be17e5b51ab7 BootID:8baf9dd7-0a0c-4fc8-920b-0fdb6e336fae Filesystems:[{Device:/var/lib/kubelet/pods/11e435f9-427b-45ce-971e-f9b184d09b07/volumes/kubernetes.io~projected/kube-api-access-v9nbb DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afaeecf6b369ab7e05ff7a979bf260e93da4c44777f124f49c2eb75881473e75/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7979b0bff7051d3e7de2ea0bf91f28e31c09df5fe190b58b55d66b3d51b9b5dc/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a57dde4-90d4-4240-b616-67e07eb614d2/volumes/kubernetes.io~projected/kube-api-access-vkqk7 DeviceMajor:0 DeviceMinor:351 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes/kubernetes.io~projected/kube-api-access-k6dkd DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1368fbe8-4942-4389-8e74-7dacb032b03e/volumes/kubernetes.io~projected/kube-api-access-drjk9 DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d52d4fd1-25a8-44d6-94d8-13757fe7790e/volumes/kubernetes.io~projected/kube-api-access-l8m9t DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8c875c2-ea97-435e-821a-067bc0d5c58e/volumes/kubernetes.io~projected/kube-api-access-tlm5f DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/664e8917-01ff-4d2c-923d-c7c65fbb5b56/volumes/kubernetes.io~projected/kube-api-access-6bs8m DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da6a00e7da63135746f3801060339751145d0c899b0a5410f6c6e5197ec569e8/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e57c8db9905549439102b4db00502eb711a1e4082222aefdd606a8f9337360e/userdata/shm DeviceMajor:0 DeviceMinor:361 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8c875c2-ea97-435e-821a-067bc0d5c58e/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bdb98a141e27e76b31379692639ff79bdb66de5dd81a4e65e63d6f94c864d53b/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a57dde4-90d4-4240-b616-67e07eb614d2/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:346 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/512f5c3d2b95a5f3a86a58b3129b890070fb21e8b90bd51f1825693e089205de/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0205d6051b2957d46025f61c2823434f601e946565b2248eb4caf1af9641af5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6fe8230-18dc-48a4-aa96-d742e707222d/volumes/kubernetes.io~projected/kube-api-access-zdknd DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cfc1709e-b549-40fc-9471-02c0d106ee9f/volumes/kubernetes.io~projected/kube-api-access-ckq4s DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57515afc6b1a5fb4cde2944ac0e2bfe93d334c3e26020b89866eae0ca6f9780f/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dcd42898-f7cc-4198-851b-db51f7ba110a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:268 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d03ba26a-59c3-47ce-988a-a4b459714565/volumes/kubernetes.io~projected/kube-api-access-sgkzv DeviceMajor:0 DeviceMinor:321 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33f0f81883389e4be510096c5e425b5cafaa34c1a94a88f4b72aa7f2043b18d5/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f37de934dda12deb8df293cfa6462d590bdc4fe22c0c67ca39c3efc38745476/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4cdfe4de8fd8d5132b3908a1d3d7a4b4c24f7fcb60dbf27c537acd393760e4f/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c7bddb76-6f22-430e-9016-ce19e959dd48/volumes/kubernetes.io~projected/kube-api-access-gl646 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5688730c6f24ee8bcc7d9d40f40fc26f099049f14af184f4c1a9add2b5ebcf1/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9160a39d465ef69ed863605fb4c1f3319a5d102edab38c03ae17f3fba0ebf194/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8c875c2-ea97-435e-821a-067bc0d5c58e/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/volumes/kubernetes.io~projected/kube-api-access-fzcns DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad08468c-b770-47bb-82b9-8c0d4d32f855/volumes/kubernetes.io~projected/kube-api-access-vpvtm DeviceMajor:0 DeviceMinor:355 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9126b699-4c1b-46ca-976c-a4e1813357a1/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eedc3daa-f8aa-4c65-a525-d5072215f3e7/volumes/kubernetes.io~projected/kube-api-access-4snqw DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c7bddb76-6f22-430e-9016-ce19e959dd48/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13b91cb8dffc8ff97a59294067359465bdeea6fcc392e9e5c8a9542f7ae216f4/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff8e353a-e360-459a-89f9-996bc1257069/volumes/kubernetes.io~projected/kube-api-access-jcphm DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c67c8d21f94be880836d9c751957db2c005ea0bcb8f22679bc6a1694d7158a7/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2ca4100-fa85-49bd-bf74-1339c6116298/volumes/kubernetes.io~projected/kube-api-access-4cxrc DeviceMajor:0 DeviceMinor:354 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16e9a2c3-c4e3-4e78-9157-926fae93f8a8/volumes/kubernetes.io~projected/kube-api-access-mv7j9 DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4253848f40f7e437c5b80082e603675b3ecceedb6fa596de5b64c88bf2d767c/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b63e01500df9d86c13bbcdf095287d4d1a740c88e9c2150df01059d92d68f624/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48738200-5d6d-4a3e-8798-700496cd1681/volumes/kubernetes.io~projected/kube-api-access-2hd86 DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d338696153e6a7b4c73119fc8fb55da0029956e7860b4568cbf3312739c84a54/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de669e87-278f-48af-bab0-625698973092/volumes/kubernetes.io~projected/kube-api-access-zvp79 DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ad4518d7c2651fa29049a332906a048239f09fceb8a9dc1a41b5ca1e623ca97/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1368fbe8-4942-4389-8e74-7dacb032b03e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d09ccbf110377bf4cecd4005bc7bd4ddede57b8388f5bbcc66579621f788f861/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/894d2520-9e05-4460-a37d-f3f8377a488b/volumes/kubernetes.io~projected/kube-api-access-g9mbx DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cafebb2b655f5e25b8577b2bd2f8c8d99b349cc9bab576fbac5f04aefb30353f/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c7bddb76-6f22-430e-9016-ce19e959dd48/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e1055486-a6b2-425f-9c20-89c93df7e080/volumes/kubernetes.io~projected/kube-api-access-z575t DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f0725d29-b4bf-4d33-8089-2c568ba41b0f/volumes/kubernetes.io~projected/kube-api-access-rd5jj DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2/volumes/kubernetes.io~projected/kube-api-access-gfhlt DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc57b913b081983f0738901b4ba4e13009779804fb419fbe114fb30ed7acbe3c/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes/kubernetes.io~projected/kube-api-access-hphfz DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f636a40a9fd1c0fce1ca6282c880b262c4e5439a5e1a14195e24ab05001fd62f/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/daa8dea664061fe4640f70d3002e4ceef9c510984edb2adfad6a0a0923cdbafa/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11e435f9-427b-45ce-971e-f9b184d09b07/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0f1aae-624b-4735-ae9b-d27b716a6d9c/volumes/kubernetes.io~projected/kube-api-access-bqpjs DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a63bf8a-50a3-4068-9835-50cfd2156350/volumes/kubernetes.io~projected/kube-api-access-vcb52 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684/volumes/kubernetes.io~projected/kube-api-access-fhtzm DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f2127f149bf105f7e6d273f1d355b361c93bebb08b3d440900c100e29b5ce20/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51924e26-cd42-4c49-9307-f041b7ab2d92/volumes/kubernetes.io~projected/kube-api-access-r5vg9 DeviceMajor:0 DeviceMinor:311 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dfa67acd059b2489cdc964514150aefa2d7bb74d2c5599b662be0502804fb85/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17a9ac1d-c1e8-49d3-9058-72d10b7f662b/volumes/kubernetes.io~projected/kube-api-access-dnd84 DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c2709bdc-ce75-411d-9554-39e7c779bf8b/volumes/kubernetes.io~projected/kube-api-access-mh62q DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0169d01-fdae-40a1-ad57-8c380c023d61/volumes/kubernetes.io~projected/kube-api-access-zzh4l DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17a9ac1d-c1e8-49d3-9058-72d10b7f662b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2e57c8db9905549 MacAddress:8a:4a:3f:55:4b:4b Speed:10000 Mtu:8901} {Name:57515afc6b1a5fb MacAddress:0a:52:17:9d:5c:cf Speed:10000 Mtu:8901} {Name:7979b0bff7051d3 MacAddress:ea:c6:e2:7f:ff:08 Speed:10000 Mtu:8901} {Name:8c67c8d21f94be8 MacAddress:26:92:76:4c:07:9f Speed:10000 Mtu:8901} {Name:8f37de934dda12d MacAddress:6e:42:98:53:3b:14 Speed:10000 Mtu:8901} {Name:afaeecf6b369ab7 MacAddress:66:5d:4c:b4:60:98 Speed:10000 Mtu:8901} {Name:b63e01500df9d86 MacAddress:8a:ce:dd:4d:2c:c9 Speed:10000 Mtu:8901} {Name:bdb98a141e27e76 MacAddress:ce:40:48:2e:54:6c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:fb:6d:3f:c7 Speed:0 Mtu:9001} {Name:br-int MacAddress:06:f9:f3:0f:cd:6c Speed:0 Mtu:8901} {Name:c4253848f40f7e4 MacAddress:f2:42:73:e8:f0:e4 Speed:10000 Mtu:8901} {Name:cafebb2b655f5e2 MacAddress:aa:52:92:86:3c:63 Speed:10000 Mtu:8901} {Name:d09ccbf110377bf MacAddress:ae:3f:57:dd:cd:f2 Speed:10000 Mtu:8901} {Name:d338696153e6a7b MacAddress:02:a7:08:5a:70:0f Speed:10000 Mtu:8901} {Name:d4cdfe4de8fd8d5 MacAddress:5a:a2:b5:f5:6c:7e Speed:10000 Mtu:8901} {Name:da6a00e7da63135 MacAddress:a2:03:90:4d:71:a1 Speed:10000 Mtu:8901} {Name:daa8dea664061fe MacAddress:c2:36:0a:7e:e4:47 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:fb:6d:3f:c7 Speed:0 Mtu:9001} {Name:f636a40a9fd1c0f MacAddress:e2:8e:a4:c3:b7:ec Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2a:db:0a:ca:37:a2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 18:44:26.091848 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.091811 5019 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 18:44:26.092056 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.091854 5019 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 18:44:26.092155 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092131 5019 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 18:44:26.092286 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092156 5019 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-179.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 18:44:26.092368 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092287 5019 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 18:44:26.092368 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092295 5019 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 18:44:26.092368 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092310 5019 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:44:26.092368 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092323 5019 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:44:26.092368 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092364 5019 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 18:44:26.092541 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092531 5019 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 18:44:26.092641 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092633 5019 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 18:44:26.092671 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092668 5019 kubelet.go:493] "Attempting to sync node with API server" Nov 03 18:44:26.092698 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092678 5019 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 18:44:26.092698 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092687 5019 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 18:44:26.092698 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092695 5019 kubelet.go:405] "Adding apiserver pod source" Nov 03 18:44:26.092778 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.092702 5019 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 18:44:26.093486 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.093472 5019 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 18:44:26.093486 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.093485 5019 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:44:26.093949 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.093929 5019 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 18:44:26.105665 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.105647 5019 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 18:44:26.105665 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.105660 5019 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:44:26.105774 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.105763 5019 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 18:44:26.105972 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.105961 5019 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 18:44:26.106009 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.105990 5019 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 18:44:26.106087 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106078 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 18:44:26.106136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106092 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 18:44:26.106136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106099 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 18:44:26.106136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106104 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 18:44:26.106136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106109 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 18:44:26.106136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106115 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 18:44:26.106136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106120 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 18:44:26.106136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106126 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 18:44:26.106136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106133 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 18:44:26.106136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106138 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 18:44:26.106511 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106147 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 18:44:26.106511 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106156 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 18:44:26.106511 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106178 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 18:44:26.106511 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.106185 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 18:44:26.109825 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.109809 5019 server.go:1263] "Started kubelet" Nov 03 18:44:26.109898 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.109850 5019 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 18:44:26.111191 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.109972 5019 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 18:44:26.111191 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.110083 5019 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 18:44:26.111191 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.110498 5019 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 18:44:26.110874 ip-10-0-139-179 systemd[1]: Started Kubernetes Kubelet. Nov 03 18:44:26.111992 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.111785 5019 server.go:317] "Adding debug handlers to kubelet server" Nov 03 18:44:26.117730 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.117435 5019 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 18:44:26.117730 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.117599 5019 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 18:44:26.117869 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.117753 5019 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:38:44 +0000 UTC" deadline="2028-04-20 06:50:50.260930241 +0000 UTC" Nov 03 18:44:26.117869 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.117805 5019 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12804h6m24.143129002s" Nov 03 18:44:26.117984 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.117902 5019 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 18:44:26.117984 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.117912 5019 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 18:44:26.118080 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.118022 5019 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 18:44:26.118168 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:26.118146 5019 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-179.us-west-2.compute.internal\" not found" Nov 03 18:44:26.119676 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.119646 5019 factory.go:55] Registering systemd factory Nov 03 18:44:26.119676 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.119668 5019 factory.go:223] Registration of the systemd container factory successfully Nov 03 18:44:26.119981 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.119964 5019 factory.go:153] Registering CRI-O factory Nov 03 18:44:26.120049 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.119984 5019 factory.go:223] Registration of the crio container factory successfully Nov 03 18:44:26.120049 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.120046 5019 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 18:44:26.120146 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.120084 5019 factory.go:103] Registering Raw factory Nov 03 18:44:26.120146 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.120098 5019 manager.go:1199] Started watching for new ooms in manager Nov 03 18:44:26.120482 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.120467 5019 manager.go:319] Starting recovery of all containers Nov 03 18:44:26.122417 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122271 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1368fbe8-4942-4389-8e74-7dacb032b03e" volumeName="kubernetes.io/secret/1368fbe8-4942-4389-8e74-7dacb032b03e-cert" seLinuxMountContext="" Nov 03 18:44:26.122417 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122309 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2" volumeName="kubernetes.io/secret/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-serving-cert" seLinuxMountContext="" Nov 03 18:44:26.122417 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122321 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" volumeName="kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-installation-pull-secrets" seLinuxMountContext="" Nov 03 18:44:26.122417 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122347 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc1709e-b549-40fc-9471-02c0d106ee9f" volumeName="kubernetes.io/configmap/cfc1709e-b549-40fc-9471-02c0d106ee9f-config-volume" seLinuxMountContext="" Nov 03 18:44:26.122417 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122366 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" volumeName="kubernetes.io/configmap/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-config" seLinuxMountContext="" Nov 03 18:44:26.122417 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122376 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eedc3daa-f8aa-4c65-a525-d5072215f3e7" volumeName="kubernetes.io/projected/eedc3daa-f8aa-4c65-a525-d5072215f3e7-kube-api-access-4snqw" seLinuxMountContext="" Nov 03 18:44:26.122417 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122387 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e435f9-427b-45ce-971e-f9b184d09b07" volumeName="kubernetes.io/secret/11e435f9-427b-45ce-971e-f9b184d09b07-serving-cert" seLinuxMountContext="" Nov 03 18:44:26.122417 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122398 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1368fbe8-4942-4389-8e74-7dacb032b03e" volumeName="kubernetes.io/projected/1368fbe8-4942-4389-8e74-7dacb032b03e-kube-api-access-drjk9" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122426 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a63bf8a-50a3-4068-9835-50cfd2156350" volumeName="kubernetes.io/configmap/9a63bf8a-50a3-4068-9835-50cfd2156350-cni-binary-copy" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122450 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a63bf8a-50a3-4068-9835-50cfd2156350" volumeName="kubernetes.io/projected/9a63bf8a-50a3-4068-9835-50cfd2156350-kube-api-access-vcb52" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122465 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f1e9ac2-4936-4d85-89ab-33cc3ef7a684" volumeName="kubernetes.io/empty-dir/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-tuned" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122477 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f1e9ac2-4936-4d85-89ab-33cc3ef7a684" volumeName="kubernetes.io/projected/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-kube-api-access-fhtzm" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122489 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc1709e-b549-40fc-9471-02c0d106ee9f" volumeName="kubernetes.io/empty-dir/cfc1709e-b549-40fc-9471-02c0d106ee9f-tmp-dir" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122500 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" volumeName="kubernetes.io/secret/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-serving-cert" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122517 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48738200-5d6d-4a3e-8798-700496cd1681" volumeName="kubernetes.io/projected/48738200-5d6d-4a3e-8798-700496cd1681-kube-api-access-2hd86" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122529 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6fe8230-18dc-48a4-aa96-d742e707222d" volumeName="kubernetes.io/projected/a6fe8230-18dc-48a4-aa96-d742e707222d-kube-api-access-zdknd" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122548 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" volumeName="kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-trusted-ca" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122563 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" volumeName="kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-kube-api-access-hphfz" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122575 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc1709e-b549-40fc-9471-02c0d106ee9f" volumeName="kubernetes.io/projected/cfc1709e-b549-40fc-9471-02c0d106ee9f-kube-api-access-ckq4s" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122586 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" volumeName="kubernetes.io/projected/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-kube-api-access-fzcns" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122598 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2ca4100-fa85-49bd-bf74-1339c6116298" volumeName="kubernetes.io/projected/d2ca4100-fa85-49bd-bf74-1339c6116298-kube-api-access-4cxrc" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122611 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0725d29-b4bf-4d33-8089-2c568ba41b0f" volumeName="kubernetes.io/projected/f0725d29-b4bf-4d33-8089-2c568ba41b0f-kube-api-access-rd5jj" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122624 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efe5bf0-abca-490f-b431-a2a72f5bbaee" volumeName="kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-certificates" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122637 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2709bdc-ce75-411d-9554-39e7c779bf8b" volumeName="kubernetes.io/configmap/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-config" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122648 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7bddb76-6f22-430e-9016-ce19e959dd48" volumeName="kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-ovnkube-config" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122660 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7bddb76-6f22-430e-9016-ce19e959dd48" volumeName="kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-ovnkube-script-lib" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122672 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d03ba26a-59c3-47ce-988a-a4b459714565" volumeName="kubernetes.io/empty-dir/d03ba26a-59c3-47ce-988a-a4b459714565-archives-path" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122685 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17a9ac1d-c1e8-49d3-9058-72d10b7f662b" volumeName="kubernetes.io/configmap/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-trusted-ca-bundle" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122703 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efe5bf0-abca-490f-b431-a2a72f5bbaee" volumeName="kubernetes.io/empty-dir/3efe5bf0-abca-490f-b431-a2a72f5bbaee-ca-trust-extracted" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122716 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9126b699-4c1b-46ca-976c-a4e1813357a1" volumeName="kubernetes.io/secret/9126b699-4c1b-46ca-976c-a4e1813357a1-agent-certs" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122728 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c875c2-ea97-435e-821a-067bc0d5c58e" volumeName="kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-default-certificate" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122740 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e435f9-427b-45ce-971e-f9b184d09b07" volumeName="kubernetes.io/projected/11e435f9-427b-45ce-971e-f9b184d09b07-kube-api-access-v9nbb" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122752 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad08468c-b770-47bb-82b9-8c0d4d32f855" volumeName="kubernetes.io/projected/ad08468c-b770-47bb-82b9-8c0d4d32f855-kube-api-access-vpvtm" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122763 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d03ba26a-59c3-47ce-988a-a4b459714565" volumeName="kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-trusted-ca-bundle" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122777 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d03ba26a-59c3-47ce-988a-a4b459714565" volumeName="kubernetes.io/projected/d03ba26a-59c3-47ce-988a-a4b459714565-kube-api-access-sgkzv" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122789 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c875c2-ea97-435e-821a-067bc0d5c58e" volumeName="kubernetes.io/projected/d8c875c2-ea97-435e-821a-067bc0d5c58e-kube-api-access-tlm5f" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122801 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1055486-a6b2-425f-9c20-89c93df7e080" volumeName="kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122813 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efe5bf0-abca-490f-b431-a2a72f5bbaee" volumeName="kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-kube-api-access-k6dkd" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122825 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efe5bf0-abca-490f-b431-a2a72f5bbaee" volumeName="kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-image-registry-private-configuration" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122839 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2" volumeName="kubernetes.io/projected/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-kube-api-access-gfhlt" seLinuxMountContext="" Nov 03 18:44:26.122843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122854 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="894d2520-9e05-4460-a37d-f3f8377a488b" volumeName="kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122865 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1055486-a6b2-425f-9c20-89c93df7e080" volumeName="kubernetes.io/projected/e1055486-a6b2-425f-9c20-89c93df7e080-kube-api-access-z575t" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122878 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51924e26-cd42-4c49-9307-f041b7ab2d92" volumeName="kubernetes.io/empty-dir/51924e26-cd42-4c49-9307-f041b7ab2d92-data-volume" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122892 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51924e26-cd42-4c49-9307-f041b7ab2d92" volumeName="kubernetes.io/projected/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-api-access-r5vg9" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122905 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17a9ac1d-c1e8-49d3-9058-72d10b7f662b" volumeName="kubernetes.io/projected/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-kube-api-access-dnd84" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122962 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efe5bf0-abca-490f-b431-a2a72f5bbaee" volumeName="kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-trusted-ca" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122978 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efe5bf0-abca-490f-b431-a2a72f5bbaee" volumeName="kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-installation-pull-secrets" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.122990 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2" volumeName="kubernetes.io/configmap/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-config" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123001 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcd42898-f7cc-4198-851b-db51f7ba110a" volumeName="kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123013 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0f1aae-624b-4735-ae9b-d27b716a6d9c" volumeName="kubernetes.io/projected/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-kube-api-access-bqpjs" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123026 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16e9a2c3-c4e3-4e78-9157-926fae93f8a8" volumeName="kubernetes.io/projected/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-kube-api-access-mv7j9" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123039 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17a9ac1d-c1e8-49d3-9058-72d10b7f662b" volumeName="kubernetes.io/empty-dir/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-tmp" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123052 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" volumeName="kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-certificates" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123064 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7bddb76-6f22-430e-9016-ce19e959dd48" volumeName="kubernetes.io/projected/c7bddb76-6f22-430e-9016-ce19e959dd48-kube-api-access-gl646" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123093 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1055486-a6b2-425f-9c20-89c93df7e080" volumeName="kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-cni-binary-copy" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123106 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a57dde4-90d4-4240-b616-67e07eb614d2" volumeName="kubernetes.io/secret/3a57dde4-90d4-4240-b616-67e07eb614d2-signing-key" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123127 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a57dde4-90d4-4240-b616-67e07eb614d2" volumeName="kubernetes.io/configmap/3a57dde4-90d4-4240-b616-67e07eb614d2-signing-cabundle" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123139 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="664e8917-01ff-4d2c-923d-c7c65fbb5b56" volumeName="kubernetes.io/projected/664e8917-01ff-4d2c-923d-c7c65fbb5b56-kube-api-access-6bs8m" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123156 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f1e9ac2-4936-4d85-89ab-33cc3ef7a684" volumeName="kubernetes.io/empty-dir/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-tmp" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123168 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0169d01-fdae-40a1-ad57-8c380c023d61" volumeName="kubernetes.io/empty-dir/b0169d01-fdae-40a1-ad57-8c380c023d61-tmp-dir" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123179 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" volumeName="kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-image-registry-private-configuration" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123192 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" volumeName="kubernetes.io/configmap/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-trusted-ca" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123203 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d52d4fd1-25a8-44d6-94d8-13757fe7790e" volumeName="kubernetes.io/configmap/d52d4fd1-25a8-44d6-94d8-13757fe7790e-serviceca" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123214 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de669e87-278f-48af-bab0-625698973092" volumeName="kubernetes.io/projected/de669e87-278f-48af-bab0-625698973092-kube-api-access-zvp79" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123228 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d94b4ea-05d0-4bbd-bf1f-3a39db705964" volumeName="kubernetes.io/configmap/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-nginx-conf" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123255 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17a9ac1d-c1e8-49d3-9058-72d10b7f662b" volumeName="kubernetes.io/empty-dir/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-snapshots" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123268 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a57dde4-90d4-4240-b616-67e07eb614d2" volumeName="kubernetes.io/projected/3a57dde4-90d4-4240-b616-67e07eb614d2-kube-api-access-vkqk7" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123282 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9126b699-4c1b-46ca-976c-a4e1813357a1" volumeName="kubernetes.io/configmap/9126b699-4c1b-46ca-976c-a4e1813357a1-konnectivity-ca" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123293 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2709bdc-ce75-411d-9554-39e7c779bf8b" volumeName="kubernetes.io/projected/c2709bdc-ce75-411d-9554-39e7c779bf8b-kube-api-access-mh62q" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123305 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d03ba26a-59c3-47ce-988a-a4b459714565" volumeName="kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-service-ca-bundle" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123318 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1055486-a6b2-425f-9c20-89c93df7e080" volumeName="kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123362 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efe5bf0-abca-490f-b431-a2a72f5bbaee" volumeName="kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-bound-sa-token" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123377 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a63bf8a-50a3-4068-9835-50cfd2156350" volumeName="kubernetes.io/configmap/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-daemon-config" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123406 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0169d01-fdae-40a1-ad57-8c380c023d61" volumeName="kubernetes.io/projected/b0169d01-fdae-40a1-ad57-8c380c023d61-kube-api-access-zzh4l" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123420 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" volumeName="kubernetes.io/empty-dir/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-ca-trust-extracted" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123435 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7bddb76-6f22-430e-9016-ce19e959dd48" volumeName="kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-env-overrides" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123448 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7bddb76-6f22-430e-9016-ce19e959dd48" volumeName="kubernetes.io/secret/c7bddb76-6f22-430e-9016-ce19e959dd48-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123460 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0725d29-b4bf-4d33-8089-2c568ba41b0f" volumeName="kubernetes.io/configmap/f0725d29-b4bf-4d33-8089-2c568ba41b0f-telemetry-config" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123488 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e435f9-427b-45ce-971e-f9b184d09b07" volumeName="kubernetes.io/configmap/11e435f9-427b-45ce-971e-f9b184d09b07-config" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123501 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17a9ac1d-c1e8-49d3-9058-72d10b7f662b" volumeName="kubernetes.io/configmap/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-service-ca-bundle" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123512 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17a9ac1d-c1e8-49d3-9058-72d10b7f662b" volumeName="kubernetes.io/secret/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-serving-cert" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123524 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" volumeName="kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-bound-sa-token" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123538 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d52d4fd1-25a8-44d6-94d8-13757fe7790e" volumeName="kubernetes.io/projected/d52d4fd1-25a8-44d6-94d8-13757fe7790e-kube-api-access-l8m9t" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123551 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c875c2-ea97-435e-821a-067bc0d5c58e" volumeName="kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-stats-auth" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123563 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff8e353a-e360-459a-89f9-996bc1257069" volumeName="kubernetes.io/projected/ff8e353a-e360-459a-89f9-996bc1257069-kube-api-access-jcphm" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123578 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0f1aae-624b-4735-ae9b-d27b716a6d9c" volumeName="kubernetes.io/configmap/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-iptables-alerter-script" seLinuxMountContext="" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123590 5019 reconstruct.go:97] "Volume reconstruction finished" Nov 03 18:44:26.124290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.123598 5019 reconciler.go:29] "Reconciler: start to sync state" Nov 03 18:44:26.131556 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.131534 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/1.log" Nov 03 18:44:26.133916 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.133898 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/1.log" Nov 03 18:44:26.140845 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.140816 5019 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 18:44:26.142254 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.142234 5019 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 18:44:26.142254 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.142255 5019 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 18:44:26.142410 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.142277 5019 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 18:44:26.142410 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:26.142320 5019 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 18:44:26.148884 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.148862 5019 generic.go:356] "Generic (PLEG): container finished" podID="9d9f7f968eecc0f1a776909177a87d50" containerID="47e1459aae180e6d1bffcf31586589104d81c833bac81511f6401f69712dca4c" exitCode=0 Nov 03 18:44:26.168593 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.168568 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/1.log" Nov 03 18:44:26.168706 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.168602 5019 generic.go:356] "Generic (PLEG): container finished" podID="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" containerID="f7f6eee515b8b8400434a0f9035f753a42c29660c2a778386f698bfae79d48cc" exitCode=255 Nov 03 18:44:26.198265 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.198237 5019 generic.go:356] "Generic (PLEG): container finished" podID="e1055486-a6b2-425f-9c20-89c93df7e080" containerID="33cc98e415c3c2a1109e54920f619cad45a89921bf3e9b70678782db06925b19" exitCode=0 Nov 03 18:44:26.198265 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.198263 5019 generic.go:356] "Generic (PLEG): container finished" podID="e1055486-a6b2-425f-9c20-89c93df7e080" containerID="2201b31a5663c0161077bfd7fae9439d237fca8e6ff0bdad1ae184e872076e9c" exitCode=0 Nov 03 18:44:26.198265 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.198271 5019 generic.go:356] "Generic (PLEG): container finished" podID="e1055486-a6b2-425f-9c20-89c93df7e080" containerID="feacae9836a9d43444a0791e81ca7dc482383a8d8b49df3d0b0cc10577743294" exitCode=0 Nov 03 18:44:26.198520 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.198277 5019 generic.go:356] "Generic (PLEG): container finished" podID="e1055486-a6b2-425f-9c20-89c93df7e080" containerID="f3cef8352fd8e6b081950409dc24d1331457e86e14f87701998ace59ed33bd58" exitCode=0 Nov 03 18:44:26.198520 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.198282 5019 generic.go:356] "Generic (PLEG): container finished" podID="e1055486-a6b2-425f-9c20-89c93df7e080" containerID="775f0f26382e3328a94b856d8408736a8a118092339c2eb791066c32c34ad254" exitCode=0 Nov 03 18:44:26.198520 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.198288 5019 generic.go:356] "Generic (PLEG): container finished" podID="e1055486-a6b2-425f-9c20-89c93df7e080" containerID="1ef10b4e12be1f840c85e2a20474546438b93143d1993e8bd37e6ad13fb2c6ed" exitCode=0 Nov 03 18:44:26.213551 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.213529 5019 manager.go:324] Recovery completed Nov 03 18:44:26.218214 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:26.218195 5019 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-179.us-west-2.compute.internal\" not found" Nov 03 18:44:26.231552 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.231532 5019 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:44:26.232698 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.232680 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:44:26.232792 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.232717 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:44:26.232792 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.232732 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:44:26.235394 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235378 5019 cpu_manager.go:226] "Starting" policy="none" Nov 03 18:44:26.235394 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235392 5019 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:44:26.235478 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235392 5019 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 18:44:26.235478 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235464 5019 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 18:44:26.235616 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235601 5019 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 18:44:26.235662 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235616 5019 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 18:44:26.235713 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235704 5019 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 18:44:26.235746 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235713 5019 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 18:44:26.235746 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235720 5019 policy_none.go:50] "Start" Nov 03 18:44:26.235746 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235730 5019 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 18:44:26.235746 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235740 5019 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 18:44:26.235882 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235872 5019 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 18:44:26.235914 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235882 5019 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 18:44:26.235914 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.235888 5019 policy_none.go:44] "Start" Nov 03 18:44:26.236913 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.236894 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:44:26.237014 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.236926 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:44:26.237014 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.236948 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:44:26.241503 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.241478 5019 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 18:44:26.243444 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:26.243424 5019 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 18:44:26.244096 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.244083 5019 manager.go:341] "Starting Device Plugin manager" Nov 03 18:44:26.244193 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:26.244113 5019 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 18:44:26.244193 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.244125 5019 server.go:87] "Starting device plugin registration server" Nov 03 18:44:26.244434 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.244421 5019 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 18:44:26.244498 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.244435 5019 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 18:44:26.244559 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.244506 5019 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 18:44:26.244641 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.244603 5019 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 18:44:26.244641 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.244637 5019 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 18:44:26.244641 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.244644 5019 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 18:44:26.245626 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.244681 5019 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T18:44:26.244627885Z","UUID":"747c2ede-cf71-45b7-96aa-119f63b697d6","Handler":null,"Name":"","Endpoint":""} Nov 03 18:44:26.246787 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.246767 5019 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 18:44:26.246901 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.246803 5019 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 18:44:26.247370 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:26.247349 5019 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 18:44:26.247453 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:26.247390 5019 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-179.us-west-2.compute.internal\" not found" Nov 03 18:44:26.296720 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.296697 5019 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 18:44:26.297152 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.297118 5019 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:44:26.297206 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.297165 5019 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:44:26.297206 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.297197 5019 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:44:26.344016 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.343957 5019 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:44:26.345186 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.345169 5019 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:44:26.346242 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.346225 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:44:26.346314 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.346255 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:44:26.346314 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.346269 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:44:26.346384 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.346316 5019 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.444604 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.444534 5019 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal"] Nov 03 18:44:26.445248 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.445208 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" event={"ID":"b1997d138e76444593b76299593de1e7","Type":"ContainerStarted","Data":"640b23f4b271b3f02d9804c1d7bd812092326f8cd39c1f801e2efcc6d8b8b3cb"} Nov 03 18:44:26.445298 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.445262 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" event={"ID":"b1997d138e76444593b76299593de1e7","Type":"ContainerStarted","Data":"2ad4518d7c2651fa29049a332906a048239f09fceb8a9dc1a41b5ca1e623ca97"} Nov 03 18:44:26.445298 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.445278 5019 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.445390 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.445363 5019 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.452073 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.452050 5019 nodeinfomanager.go:471] existing CSINode "ip-10-0-139-179.us-west-2.compute.internal" is owned by different node (oldNodeID="0bc97d9e-8430-4184-96fc-7e8a882299d8", newNodeID=""), cleaning up... Nov 03 18:44:26.452817 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.452801 5019 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.452865 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.452857 5019 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.453433 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.453417 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeNotReady" Nov 03 18:44:26.453494 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.453444 5019 setters.go:546] "Node became not ready" node="ip-10-0-139-179.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T18:44:26Z","lastTransitionTime":"2026-11-03T18:44:26Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 18:44:26.502688 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.502661 5019 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:44:26.502809 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:26.502718 5019 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.503436 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.503422 5019 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:44:26.503474 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:26.503458 5019 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.597616 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.597531 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d9f7f968eecc0f1a776909177a87d50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"9d9f7f968eecc0f1a776909177a87d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.597616 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.597596 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d9f7f968eecc0f1a776909177a87d50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"9d9f7f968eecc0f1a776909177a87d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.597794 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.597626 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1997d138e76444593b76299593de1e7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"b1997d138e76444593b76299593de1e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.698470 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.698437 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d9f7f968eecc0f1a776909177a87d50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"9d9f7f968eecc0f1a776909177a87d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.698641 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.698508 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d9f7f968eecc0f1a776909177a87d50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"9d9f7f968eecc0f1a776909177a87d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.698641 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.698529 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1997d138e76444593b76299593de1e7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"b1997d138e76444593b76299593de1e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.698641 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.698572 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1997d138e76444593b76299593de1e7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"b1997d138e76444593b76299593de1e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.698641 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.698562 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d9f7f968eecc0f1a776909177a87d50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"9d9f7f968eecc0f1a776909177a87d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:26.698641 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:26.698625 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d9f7f968eecc0f1a776909177a87d50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal\" (UID: \"9d9f7f968eecc0f1a776909177a87d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:27.093605 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.093573 5019 apiserver.go:52] "Watching apiserver" Nov 03 18:44:27.203182 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.203148 5019 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:27.238609 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.238580 5019 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 18:44:27.239308 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.239274 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-nrx4t","openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722","kube-system/kas-connection-checker-685895c7c9-ht2s6","openshift-ovn-kubernetes/ovnkube-node-8ptts","openshift-network-diagnostics/network-check-target-g2vhp","openshift-dns/node-resolver-dlbxv","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4","kube-system/global-pull-secret-syncer-fn8l9","openshift-image-registry/image-registry-67986d9d7f-m6wxv","openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt","openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx","openshift-multus/network-metrics-daemon-l7j4q","openshift-console-operator/console-operator-7cd56b8dbc-48wh4","openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg","openshift-insights/insights-operator-7c4986d449-plzjk","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-179.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-bqhww","openshift-ingress/router-default-8dd798ffc-wswht","kube-system/konnectivity-agent-dwp5g","openshift-image-registry/node-ca-sncd9","kube-system/kas-connection-checker-685895c7c9-gjwzc","openshift-network-operator/iptables-alerter-5skrq","kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal","openshift-image-registry/image-registry-78c7cd5678-n2phb","openshift-ingress-canary/ingress-canary-d8l56","openshift-service-ca/service-ca-5fffc97bdf-mwz7s","openshift-dns/dns-default-9pzjb","openshift-multus/multus-rwwj5","openshift-insights/insights-runtime-extractor-w77sj","openshift-network-console/networking-console-plugin-86c478bb88-9q8tt","openshift-insights/periodic-gathering-7nsd9-jcfjp","kube-system/kas-connection-checker-685895c7c9-l6xkx","openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk"] Nov 03 18:44:27.241210 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.241188 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:27.241564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.241547 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:27.241878 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.241861 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.242483 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.242466 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:27.243804 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.243786 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:27.244192 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.244172 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:27.251401 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.251382 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.252836 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.252817 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:27.253923 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.253909 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.256496 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.256479 5019 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:44:27.256570 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.256521 5019 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal" Nov 03 18:44:27.289489 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.289459 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 18:44:27.289489 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.289475 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.289713 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.289519 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.289713 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.289665 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.289713 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.289679 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 18:44:27.289820 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.289713 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 18:44:27.290291 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.290271 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.290716 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.290688 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.290790 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.290768 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 18:44:27.290790 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.290783 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 18:44:27.290875 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.290785 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.290875 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.290801 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.290875 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.290831 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 18:44:27.291016 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.290942 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.291016 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.290963 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 18:44:27.291117 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.291031 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 18:44:27.291117 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.291094 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 18:44:27.291198 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.291119 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.291198 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.291173 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 18:44:27.291303 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.291196 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 18:44:27.291303 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.291206 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 18:44:27.291303 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.291227 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.291303 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.291231 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 18:44:27.291303 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.291249 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.291586 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.291544 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 18:44:27.291958 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.291941 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 18:44:27.300435 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.300414 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 18:44:27.300529 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.300515 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 18:44:27.303397 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.303382 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 18:44:27.320561 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.320542 5019 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 18:44:27.335648 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.335626 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-grblk\"" Nov 03 18:44:27.335730 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.335701 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k4rz5\"" Nov 03 18:44:27.336251 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.336236 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 18:44:27.336314 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.336264 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bctnd\"" Nov 03 18:44:27.336314 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.336287 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 18:44:27.336590 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.336571 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7r6fd\"" Nov 03 18:44:27.336701 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.336624 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 18:44:27.336701 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.336645 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.336701 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.336647 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.336701 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.336675 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 18:44:27.337131 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.337118 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8n5dg\"" Nov 03 18:44:27.337195 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.337188 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jwgdr\"" Nov 03 18:44:27.337854 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.337837 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 18:44:27.337854 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.337837 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 18:44:27.338051 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.337868 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.338051 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.337869 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.338051 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.337906 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 18:44:27.338051 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.337948 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.338051 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.337998 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 18:44:27.338335 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338111 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 18:44:27.338335 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338190 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.338335 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338259 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 18:44:27.338793 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338769 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 18:44:27.338883 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338778 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.338883 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338788 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.338883 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338796 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.338883 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338848 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.338883 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338869 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.338883 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338882 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.339155 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338893 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.339155 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338909 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.339155 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338939 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.339155 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.338964 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.339155 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.339050 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.340078 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.340046 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 18:44:27.340485 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.340464 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 18:44:27.350400 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.350349 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 18:44:27.350477 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.350453 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 18:44:27.389244 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389226 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 18:44:27.389244 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389233 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.389244 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389234 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 18:44:27.389478 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389418 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.389478 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389423 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.389563 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389486 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.389563 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389505 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.389563 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389493 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.389563 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389536 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.389726 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389566 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 18:44:27.389726 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389583 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.389726 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389599 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.389726 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389617 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 18:44:27.389726 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389626 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 18:44:27.389726 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389632 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 18:44:27.389726 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389644 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 18:44:27.389726 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389634 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.390005 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389787 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 18:44:27.390005 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389801 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 18:44:27.390005 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389853 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 18:44:27.390005 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389917 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 18:44:27.390005 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389960 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 18:44:27.390005 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.389962 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 18:44:27.390263 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390009 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 18:44:27.390263 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390037 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gq4wc\"" Nov 03 18:44:27.390263 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390096 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-l6xrr\"" Nov 03 18:44:27.390263 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390156 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kqw67\"" Nov 03 18:44:27.390263 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390179 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d8dtx\"" Nov 03 18:44:27.390471 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390314 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 18:44:27.390471 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390387 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qnhrd\"" Nov 03 18:44:27.390471 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390387 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45nsg\"" Nov 03 18:44:27.390471 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390401 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-97rln\"" Nov 03 18:44:27.390471 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390408 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4xp5l\"" Nov 03 18:44:27.390471 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390430 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hsjrx\"" Nov 03 18:44:27.390471 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390464 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q46mt\"" Nov 03 18:44:27.390693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390497 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sjcxp\"" Nov 03 18:44:27.390693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390560 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rw2l5\"" Nov 03 18:44:27.390693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.390587 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-99m4h\"" Nov 03 18:44:27.402639 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.402622 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-tmp\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:27.402706 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.402649 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhlt\" (UniqueName: \"kubernetes.io/projected/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-kube-api-access-gfhlt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:27.402706 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.402670 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-installation-pull-secrets\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.402706 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.402689 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-sys-fs\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.402845 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.402713 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-trusted-ca\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.402845 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.402736 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snqw\" (UniqueName: \"kubernetes.io/projected/eedc3daa-f8aa-4c65-a525-d5072215f3e7-kube-api-access-4snqw\") pod \"kas-connection-checker-685895c7c9-l6xkx\" (UID: \"eedc3daa-f8aa-4c65-a525-d5072215f3e7\") " pod="kube-system/kas-connection-checker-685895c7c9-l6xkx" Nov 03 18:44:27.402845 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.402761 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-log-socket\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.402845 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.402800 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-tmp\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:27.403065 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.402861 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-cni-netd\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.403065 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.402900 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:27.403065 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.402967 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:27.403065 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.402969 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-installation-pull-secrets\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.403065 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403014 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.403248 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403116 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:27.403248 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403145 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:27.403248 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403150 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc1709e-b549-40fc-9471-02c0d106ee9f-tmp-dir\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:27.403248 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403212 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc1709e-b549-40fc-9471-02c0d106ee9f-tmp-dir\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:27.403248 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403233 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-netns\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.403483 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403263 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-socket-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.403483 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403309 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:27.403483 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403316 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:27.403483 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403350 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-trusted-ca\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.403483 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403361 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-cni-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.403483 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403466 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-cni-multus\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.403692 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403493 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-var-lib-kubelet\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.403692 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403512 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0169d01-fdae-40a1-ad57-8c380c023d61-tmp-dir\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:44:27.403692 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403553 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:27.403692 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403558 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0169d01-fdae-40a1-ad57-8c380c023d61-tmp-dir\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:44:27.403692 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403586 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bs8m\" (UniqueName: \"kubernetes.io/projected/664e8917-01ff-4d2c-923d-c7c65fbb5b56-kube-api-access-6bs8m\") pod \"network-check-source-6f78d59f99-mzf7t\" (UID: \"664e8917-01ff-4d2c-923d-c7c65fbb5b56\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t" Nov 03 18:44:27.403692 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403618 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-ovnkube-config\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.403692 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403619 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:27.403919 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403695 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-default-certificate\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:27.403919 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403743 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-conf-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.403919 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403833 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-kubernetes\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.403919 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403847 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-ovnkube-config\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.403919 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403851 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z575t\" (UniqueName: \"kubernetes.io/projected/e1055486-a6b2-425f-9c20-89c93df7e080-kube-api-access-z575t\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.403919 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403883 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:27.404104 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403919 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcd42898-f7cc-4198-851b-db51f7ba110a-kubelet-config\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:27.404104 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403928 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-default-certificate\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:27.404104 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.403960 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-certificates\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.404104 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404090 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-systemd-units\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.404235 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404104 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:27.404235 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404107 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-certificates\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.404235 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404120 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-slash\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.404235 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404153 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-env-overrides\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.404235 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404180 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-kubelet\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.404454 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404269 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:27.404454 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404269 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-env-overrides\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.404454 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404306 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-lib-modules\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.404454 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404346 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dkd\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-kube-api-access-k6dkd\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.404454 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404364 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11e435f9-427b-45ce-971e-f9b184d09b07-serving-cert\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:27.404454 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404416 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-sys\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.404643 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404519 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-host-slash\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:44:27.404643 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404520 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11e435f9-427b-45ce-971e-f9b184d09b07-serving-cert\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:27.404643 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404553 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.404643 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404575 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-ovn\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.404643 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404599 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-stats-auth\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:27.404643 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404637 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-serving-cert\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:27.404828 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404733 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgkzv\" (UniqueName: \"kubernetes.io/projected/d03ba26a-59c3-47ce-988a-a4b459714565-kube-api-access-sgkzv\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:27.404828 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404761 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-installation-pull-secrets\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.404828 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404779 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-stats-auth\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:27.404828 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404781 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51924e26-cd42-4c49-9307-f041b7ab2d92-data-volume\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.404960 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404842 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51924e26-cd42-4c49-9307-f041b7ab2d92-data-volume\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.404993 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404976 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-installation-pull-secrets\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.404993 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.404976 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-host\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.405060 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405011 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1368fbe8-4942-4389-8e74-7dacb032b03e-cert\") pod \"ingress-canary-d8l56\" (UID: \"1368fbe8-4942-4389-8e74-7dacb032b03e\") " pod="openshift-ingress-canary/ingress-canary-d8l56" Nov 03 18:44:27.405060 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405035 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9126b699-4c1b-46ca-976c-a4e1813357a1-agent-certs\") pod \"konnectivity-agent-dwp5g\" (UID: \"9126b699-4c1b-46ca-976c-a4e1813357a1\") " pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:44:27.405060 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405045 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-serving-cert\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:27.405219 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405204 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1368fbe8-4942-4389-8e74-7dacb032b03e-cert\") pod \"ingress-canary-d8l56\" (UID: \"1368fbe8-4942-4389-8e74-7dacb032b03e\") " pod="openshift-ingress-canary/ingress-canary-d8l56" Nov 03 18:44:27.405263 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405237 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-ca-trust-extracted\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.405263 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405253 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-modprobe-d\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.405323 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405268 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-run-netns\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.405323 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405284 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-service-ca-bundle\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:27.405323 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405297 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-run\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.405452 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405369 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9126b699-4c1b-46ca-976c-a4e1813357a1-agent-certs\") pod \"konnectivity-agent-dwp5g\" (UID: \"9126b699-4c1b-46ca-976c-a4e1813357a1\") " pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:44:27.405452 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405371 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-cni-bin\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.405452 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405399 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-ca-trust-extracted\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.405452 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405407 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-service-ca-bundle\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:27.405452 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405416 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkqk7\" (UniqueName: \"kubernetes.io/projected/3a57dde4-90d4-4240-b616-67e07eb614d2-kube-api-access-vkqk7\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:27.405452 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405445 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.405693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405525 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-registration-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.405693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405542 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhtzm\" (UniqueName: \"kubernetes.io/projected/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-kube-api-access-fhtzm\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.405693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405559 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd5jj\" (UniqueName: \"kubernetes.io/projected/f0725d29-b4bf-4d33-8089-2c568ba41b0f-kube-api-access-rd5jj\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:27.405693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405574 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckq4s\" (UniqueName: \"kubernetes.io/projected/cfc1709e-b549-40fc-9471-02c0d106ee9f-kube-api-access-ckq4s\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:27.405693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405617 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.405693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405671 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:27.406023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405712 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mbx\" (UniqueName: \"kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx\") pod \"network-check-target-g2vhp\" (UID: \"894d2520-9e05-4460-a37d-f3f8377a488b\") " pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:27.406023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405775 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5vg9\" (UniqueName: \"kubernetes.io/projected/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-api-access-r5vg9\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.406023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405813 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:27.406023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405843 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.406023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405869 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-tuned\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.406023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405893 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8m9t\" (UniqueName: \"kubernetes.io/projected/d52d4fd1-25a8-44d6-94d8-13757fe7790e-kube-api-access-l8m9t\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:44:27.406023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405918 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-var-lib-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.406023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405951 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-trusted-ca-bundle\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:27.406023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405952 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-tuned\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.406023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.405975 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.406023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406002 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcb52\" (UniqueName: \"kubernetes.io/projected/9a63bf8a-50a3-4068-9835-50cfd2156350-kube-api-access-vcb52\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.406023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406022 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysctl-d\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406041 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-systemd\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406061 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc1709e-b549-40fc-9471-02c0d106ee9f-config-volume\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406097 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlm5f\" (UniqueName: \"kubernetes.io/projected/d8c875c2-ea97-435e-821a-067bc0d5c58e-kube-api-access-tlm5f\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406188 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcd42898-f7cc-4198-851b-db51f7ba110a-dbus\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406231 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-trusted-ca-bundle\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406228 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-certificates\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406251 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406296 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d52d4fd1-25a8-44d6-94d8-13757fe7790e-serviceca\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406364 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-cni-binary-copy\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406385 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh62q\" (UniqueName: \"kubernetes.io/projected/c2709bdc-ce75-411d-9554-39e7c779bf8b-kube-api-access-mh62q\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406403 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpvtm\" (UniqueName: \"kubernetes.io/projected/ad08468c-b770-47bb-82b9-8c0d4d32f855-kube-api-access-vpvtm\") pod \"migrator-66df5d496b-tqfcb\" (UID: \"ad08468c-b770-47bb-82b9-8c0d4d32f855\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406411 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-certificates\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406420 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnd84\" (UniqueName: \"kubernetes.io/projected/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-kube-api-access-dnd84\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406442 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvp79\" (UniqueName: \"kubernetes.io/projected/de669e87-278f-48af-bab0-625698973092-kube-api-access-zvp79\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406451 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc1709e-b549-40fc-9471-02c0d106ee9f-config-volume\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406493 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406515 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-systemd\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406539 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3efe5bf0-abca-490f-b431-a2a72f5bbaee-ca-trust-extracted\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.406564 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406565 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-bound-sa-token\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406572 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-cni-binary-copy\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406606 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl646\" (UniqueName: \"kubernetes.io/projected/c7bddb76-6f22-430e-9016-ce19e959dd48-kube-api-access-gl646\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406633 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d52d4fd1-25a8-44d6-94d8-13757fe7790e-serviceca\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406660 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3efe5bf0-abca-490f-b431-a2a72f5bbaee-ca-trust-extracted\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406640 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcphm\" (UniqueName: \"kubernetes.io/projected/ff8e353a-e360-459a-89f9-996bc1257069-kube-api-access-jcphm\") pod \"volume-data-source-validator-58c9f5f46d-vrjn4\" (UID: \"ff8e353a-e360-459a-89f9-996bc1257069\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406701 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-k8s-cni-cncf-io\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406725 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-config\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406766 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d52d4fd1-25a8-44d6-94d8-13757fe7790e-host\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406799 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzh4l\" (UniqueName: \"kubernetes.io/projected/b0169d01-fdae-40a1-ad57-8c380c023d61-kube-api-access-zzh4l\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406817 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-service-ca-bundle\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406851 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-hostroot\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406872 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-config\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406945 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-service-ca-bundle\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406963 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3a57dde4-90d4-4240-b616-67e07eb614d2-signing-key\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.406993 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysconfig\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407021 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f0725d29-b4bf-4d33-8089-2c568ba41b0f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407117 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hd86\" (UniqueName: \"kubernetes.io/projected/48738200-5d6d-4a3e-8798-700496cd1681-kube-api-access-2hd86\") pod \"kas-connection-checker-685895c7c9-gjwzc\" (UID: \"48738200-5d6d-4a3e-8798-700496cd1681\") " pod="kube-system/kas-connection-checker-685895c7c9-gjwzc" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407140 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407155 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3a57dde4-90d4-4240-b616-67e07eb614d2-signing-key\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407159 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cxrc\" (UniqueName: \"kubernetes.io/projected/d2ca4100-fa85-49bd-bf74-1339c6116298-kube-api-access-4cxrc\") pod \"migrator-66df5d496b-jnnvk\" (UID: \"d2ca4100-fa85-49bd-bf74-1339c6116298\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407212 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407243 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-cnibin\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407266 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407271 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f0725d29-b4bf-4d33-8089-2c568ba41b0f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407314 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqpjs\" (UniqueName: \"kubernetes.io/projected/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-kube-api-access-bqpjs\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407358 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-etc-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407400 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysctl-conf\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.407399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407405 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407427 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-tmp\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407462 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-kubelet\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407487 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3a57dde4-90d4-4240-b616-67e07eb614d2-signing-cabundle\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407506 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-device-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407515 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-tmp\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407538 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407568 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9126b699-4c1b-46ca-976c-a4e1813357a1-konnectivity-ca\") pod \"konnectivity-agent-dwp5g\" (UID: \"9126b699-4c1b-46ca-976c-a4e1813357a1\") " pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407596 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-socket-dir-parent\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407654 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-multus-certs\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407682 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0169d01-fdae-40a1-ad57-8c380c023d61-hosts-file\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407692 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3a57dde4-90d4-4240-b616-67e07eb614d2-signing-cabundle\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407710 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjk9\" (UniqueName: \"kubernetes.io/projected/1368fbe8-4942-4389-8e74-7dacb032b03e-kube-api-access-drjk9\") pod \"ingress-canary-d8l56\" (UID: \"1368fbe8-4942-4389-8e74-7dacb032b03e\") " pod="openshift-ingress-canary/ingress-canary-d8l56" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407732 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407745 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9126b699-4c1b-46ca-976c-a4e1813357a1-konnectivity-ca\") pod \"konnectivity-agent-dwp5g\" (UID: \"9126b699-4c1b-46ca-976c-a4e1813357a1\") " pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407792 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11e435f9-427b-45ce-971e-f9b184d09b07-config\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407859 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407899 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407927 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-system-cni-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407953 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407970 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11e435f9-427b-45ce-971e-f9b184d09b07-config\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.407988 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv7j9\" (UniqueName: \"kubernetes.io/projected/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-kube-api-access-mv7j9\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408054 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-bound-sa-token\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408076 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-ovnkube-script-lib\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408093 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-snapshots\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408097 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1055486-a6b2-425f-9c20-89c93df7e080-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408108 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-system-cni-dir\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408136 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcns\" (UniqueName: \"kubernetes.io/projected/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-kube-api-access-fzcns\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408163 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdknd\" (UniqueName: \"kubernetes.io/projected/a6fe8230-18dc-48a4-aa96-d742e707222d-kube-api-access-zdknd\") pod \"kas-connection-checker-685895c7c9-ht2s6\" (UID: \"a6fe8230-18dc-48a4-aa96-d742e707222d\") " pod="kube-system/kas-connection-checker-685895c7c9-ht2s6" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408188 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9nbb\" (UniqueName: \"kubernetes.io/projected/11e435f9-427b-45ce-971e-f9b184d09b07-kube-api-access-v9nbb\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408214 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-snapshots\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408219 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408244 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-cnibin\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408271 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-etc-kubernetes\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408242 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcd42898-f7cc-4198-851b-db51f7ba110a-original-pull-secret\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408306 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-serving-cert\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408312 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7bddb76-6f22-430e-9016-ce19e959dd48-ovnkube-script-lib\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408382 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-image-registry-private-configuration\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408419 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphfz\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-kube-api-access-hphfz\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408444 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408445 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-node-log\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408482 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d03ba26a-59c3-47ce-988a-a4b459714565-archives-path\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408510 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-serving-cert\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408517 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-cni-bin\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408540 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d03ba26a-59c3-47ce-988a-a4b459714565-archives-path\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408545 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-image-registry-private-configuration\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408557 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-daemon-config\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.408566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408584 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-iptables-alerter-script\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:44:27.410029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408607 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51924e26-cd42-4c49-9307-f041b7ab2d92-crio-socket\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.410029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408748 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a63bf8a-50a3-4068-9835-50cfd2156350-cni-binary-copy\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.410029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408820 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-daemon-config\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.410029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408835 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-trusted-ca\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.410029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408863 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-image-registry-private-configuration\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.410029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408881 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a63bf8a-50a3-4068-9835-50cfd2156350-cni-binary-copy\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.410029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408882 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.410029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408922 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7bddb76-6f22-430e-9016-ce19e959dd48-ovn-node-metrics-cert\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.410029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.408971 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-iptables-alerter-script\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:44:27.410029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.409030 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7bddb76-6f22-430e-9016-ce19e959dd48-ovn-node-metrics-cert\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.410029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.409155 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-image-registry-private-configuration\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.410029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.409185 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-trusted-ca\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.459014 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.458986 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snqw\" (UniqueName: \"kubernetes.io/projected/eedc3daa-f8aa-4c65-a525-d5072215f3e7-kube-api-access-4snqw\") pod \"kas-connection-checker-685895c7c9-l6xkx\" (UID: \"eedc3daa-f8aa-4c65-a525-d5072215f3e7\") " pod="kube-system/kas-connection-checker-685895c7c9-l6xkx" Nov 03 18:44:27.459255 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.459234 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z575t\" (UniqueName: \"kubernetes.io/projected/e1055486-a6b2-425f-9c20-89c93df7e080-kube-api-access-z575t\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.459373 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.459359 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bs8m\" (UniqueName: \"kubernetes.io/projected/664e8917-01ff-4d2c-923d-c7c65fbb5b56-kube-api-access-6bs8m\") pod \"network-check-source-6f78d59f99-mzf7t\" (UID: \"664e8917-01ff-4d2c-923d-c7c65fbb5b56\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mzf7t" Nov 03 18:44:27.460723 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.460702 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkqk7\" (UniqueName: \"kubernetes.io/projected/3a57dde4-90d4-4240-b616-67e07eb614d2-kube-api-access-vkqk7\") pod \"service-ca-5fffc97bdf-mwz7s\" (UID: \"3a57dde4-90d4-4240-b616-67e07eb614d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mwz7s" Nov 03 18:44:27.460796 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.460754 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6dkd\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-kube-api-access-k6dkd\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.460878 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.460862 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfhlt\" (UniqueName: \"kubernetes.io/projected/5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2-kube-api-access-gfhlt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4\" (UID: \"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" Nov 03 18:44:27.461001 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.460985 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgkzv\" (UniqueName: \"kubernetes.io/projected/d03ba26a-59c3-47ce-988a-a4b459714565-kube-api-access-sgkzv\") pod \"periodic-gathering-7nsd9-jcfjp\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:44:27.465893 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.465877 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckq4s\" (UniqueName: \"kubernetes.io/projected/cfc1709e-b549-40fc-9471-02c0d106ee9f-kube-api-access-ckq4s\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:27.466121 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.466103 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd5jj\" (UniqueName: \"kubernetes.io/projected/f0725d29-b4bf-4d33-8089-2c568ba41b0f-kube-api-access-rd5jj\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:27.466690 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.466674 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9mbx\" (UniqueName: \"kubernetes.io/projected/894d2520-9e05-4460-a37d-f3f8377a488b-kube-api-access-g9mbx\") pod \"network-check-target-g2vhp\" (UID: \"894d2520-9e05-4460-a37d-f3f8377a488b\") " pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:27.470274 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.470257 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl646\" (UniqueName: \"kubernetes.io/projected/c7bddb76-6f22-430e-9016-ce19e959dd48-kube-api-access-gl646\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.470988 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.470969 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhtzm\" (UniqueName: \"kubernetes.io/projected/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-kube-api-access-fhtzm\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.471241 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.471215 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlm5f\" (UniqueName: \"kubernetes.io/projected/d8c875c2-ea97-435e-821a-067bc0d5c58e-kube-api-access-tlm5f\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:27.471361 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.471315 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcb52\" (UniqueName: \"kubernetes.io/projected/9a63bf8a-50a3-4068-9835-50cfd2156350-kube-api-access-vcb52\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.471515 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.471500 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh62q\" (UniqueName: \"kubernetes.io/projected/c2709bdc-ce75-411d-9554-39e7c779bf8b-kube-api-access-mh62q\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:27.472671 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.472650 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8m9t\" (UniqueName: \"kubernetes.io/projected/d52d4fd1-25a8-44d6-94d8-13757fe7790e-kube-api-access-l8m9t\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:44:27.472768 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.472670 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpvtm\" (UniqueName: \"kubernetes.io/projected/ad08468c-b770-47bb-82b9-8c0d4d32f855-kube-api-access-vpvtm\") pod \"migrator-66df5d496b-tqfcb\" (UID: \"ad08468c-b770-47bb-82b9-8c0d4d32f855\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb" Nov 03 18:44:27.472884 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.472817 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdknd\" (UniqueName: \"kubernetes.io/projected/a6fe8230-18dc-48a4-aa96-d742e707222d-kube-api-access-zdknd\") pod \"kas-connection-checker-685895c7c9-ht2s6\" (UID: \"a6fe8230-18dc-48a4-aa96-d742e707222d\") " pod="kube-system/kas-connection-checker-685895c7c9-ht2s6" Nov 03 18:44:27.472991 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.472891 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hd86\" (UniqueName: \"kubernetes.io/projected/48738200-5d6d-4a3e-8798-700496cd1681-kube-api-access-2hd86\") pod \"kas-connection-checker-685895c7c9-gjwzc\" (UID: \"48738200-5d6d-4a3e-8798-700496cd1681\") " pod="kube-system/kas-connection-checker-685895c7c9-gjwzc" Nov 03 18:44:27.472991 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.472950 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5vg9\" (UniqueName: \"kubernetes.io/projected/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-api-access-r5vg9\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.473142 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.473121 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnd84\" (UniqueName: \"kubernetes.io/projected/17a9ac1d-c1e8-49d3-9058-72d10b7f662b-kube-api-access-dnd84\") pod \"insights-operator-7c4986d449-plzjk\" (UID: \"17a9ac1d-c1e8-49d3-9058-72d10b7f662b\") " pod="openshift-insights/insights-operator-7c4986d449-plzjk" Nov 03 18:44:27.475675 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.475653 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cxrc\" (UniqueName: \"kubernetes.io/projected/d2ca4100-fa85-49bd-bf74-1339c6116298-kube-api-access-4cxrc\") pod \"migrator-66df5d496b-jnnvk\" (UID: \"d2ca4100-fa85-49bd-bf74-1339c6116298\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk" Nov 03 18:44:27.476715 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.476663 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvp79\" (UniqueName: \"kubernetes.io/projected/de669e87-278f-48af-bab0-625698973092-kube-api-access-zvp79\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:27.476857 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.476838 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcphm\" (UniqueName: \"kubernetes.io/projected/ff8e353a-e360-459a-89f9-996bc1257069-kube-api-access-jcphm\") pod \"volume-data-source-validator-58c9f5f46d-vrjn4\" (UID: \"ff8e353a-e360-459a-89f9-996bc1257069\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vrjn4" Nov 03 18:44:27.476945 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.476862 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzh4l\" (UniqueName: \"kubernetes.io/projected/b0169d01-fdae-40a1-ad57-8c380c023d61-kube-api-access-zzh4l\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:44:27.477106 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.477085 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv7j9\" (UniqueName: \"kubernetes.io/projected/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-kube-api-access-mv7j9\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.477383 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.477314 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-bound-sa-token\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.477478 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.477437 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphfz\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-kube-api-access-hphfz\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.477703 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.477681 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9nbb\" (UniqueName: \"kubernetes.io/projected/11e435f9-427b-45ce-971e-f9b184d09b07-kube-api-access-v9nbb\") pod \"service-ca-operator-544d749c8c-5d722\" (UID: \"11e435f9-427b-45ce-971e-f9b184d09b07\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" Nov 03 18:44:27.477788 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.477702 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcns\" (UniqueName: \"kubernetes.io/projected/d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1-kube-api-access-fzcns\") pod \"console-operator-7cd56b8dbc-48wh4\" (UID: \"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:27.477867 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.477849 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqpjs\" (UniqueName: \"kubernetes.io/projected/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-kube-api-access-bqpjs\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:44:27.478033 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.478015 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-bound-sa-token\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.478096 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.478081 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjk9\" (UniqueName: \"kubernetes.io/projected/1368fbe8-4942-4389-8e74-7dacb032b03e-kube-api-access-drjk9\") pod \"ingress-canary-d8l56\" (UID: \"1368fbe8-4942-4389-8e74-7dacb032b03e\") " pod="openshift-ingress-canary/ingress-canary-d8l56" Nov 03 18:44:27.509737 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509718 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.509849 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509744 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-systemd\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.509849 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509765 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-k8s-cni-cncf-io\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.509849 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509781 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d52d4fd1-25a8-44d6-94d8-13757fe7790e-host\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:44:27.509849 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509797 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-hostroot\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.509849 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509817 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysconfig\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.509849 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509844 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509846 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509857 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-k8s-cni-cncf-io\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509869 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-cnibin\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509857 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-systemd\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509877 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d52d4fd1-25a8-44d6-94d8-13757fe7790e-host\") pod \"node-ca-sncd9\" (UID: \"d52d4fd1-25a8-44d6-94d8-13757fe7790e\") " pod="openshift-image-registry/node-ca-sncd9" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509880 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-hostroot\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509907 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509935 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-etc-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509943 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-cnibin\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.509910 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysconfig\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.509974 5019 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510051 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-etc-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510054 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysctl-conf\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.510111 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.510086 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert podName:0d94b4ea-05d0-4bbd-bf1f-3a39db705964 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:28.010067609 +0000 UTC m=+1.982628786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9q8tt" (UID: "0d94b4ea-05d0-4bbd-bf1f-3a39db705964") : secret "networking-console-plugin-cert" not found Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510142 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-kubelet\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510165 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-device-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510172 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysctl-conf\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510190 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510195 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-kubelet\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510217 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-socket-dir-parent\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510233 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-device-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510241 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-multus-certs\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510259 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510231 5019 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510277 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-multus-certs\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510305 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-socket-dir-parent\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510314 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0169d01-fdae-40a1-ad57-8c380c023d61-hosts-file\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510282 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0169d01-fdae-40a1-ad57-8c380c023d61-hosts-file\") pod \"node-resolver-dlbxv\" (UID: \"b0169d01-fdae-40a1-ad57-8c380c023d61\") " pod="openshift-dns/node-resolver-dlbxv" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510374 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510393 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510427 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-system-cni-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510456 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-system-cni-dir\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510484 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-cnibin\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510508 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-etc-kubernetes\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510520 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-system-cni-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510520 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-system-cni-dir\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510538 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-cnibin\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510544 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-etc-kubernetes\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510535 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-node-log\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.510486 5019 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510567 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-node-log\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.510540 5019 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510587 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-cni-bin\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.510617 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls podName:c2709bdc-ce75-411d-9554-39e7c779bf8b nodeName:}" failed. No retries permitted until 2026-11-03 18:44:28.01060199 +0000 UTC m=+1.983163151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-txhfg" (UID: "c2709bdc-ce75-411d-9554-39e7c779bf8b") : secret "samples-operator-tls" not found Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.510648 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls podName:51924e26-cd42-4c49-9307-f041b7ab2d92 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:28.010631657 +0000 UTC m=+1.983192825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w77sj" (UID: "51924e26-cd42-4c49-9307-f041b7ab2d92") : secret "insights-runtime-extractor-tls" not found Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510690 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51924e26-cd42-4c49-9307-f041b7ab2d92-crio-socket\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510720 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510694 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-cni-bin\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510751 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-sys-fs\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.510766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510770 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51924e26-cd42-4c49-9307-f041b7ab2d92-crio-socket\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510796 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-log-socket\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510823 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-cni-netd\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510827 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-sys-fs\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510832 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-log-socket\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510853 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510872 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510871 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-cni-netd\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510890 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-netns\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510910 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510917 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-socket-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510941 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-cni-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510950 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-run-netns\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510964 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-cni-multus\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.510983 5019 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.510988 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-var-lib-kubelet\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511003 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-socket-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511018 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511039 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-cni-multus\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511045 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-conf-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511040 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-cni-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511068 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-kubernetes\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511081 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-var-lib-kubelet\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511089 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcd42898-f7cc-4198-851b-db51f7ba110a-kubelet-config\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.511102 5019 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511121 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-systemd-units\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511131 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcd42898-f7cc-4198-851b-db51f7ba110a-kubelet-config\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511141 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-multus-conf-dir\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511151 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-slash\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511155 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-kubernetes\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511171 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-systemd-units\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511178 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-slash\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511178 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-kubelet\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.511187 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls podName:f0725d29-b4bf-4d33-8089-2c568ba41b0f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:28.011174169 +0000 UTC m=+1.983735340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4dlvx" (UID: "f0725d29-b4bf-4d33-8089-2c568ba41b0f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511210 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a63bf8a-50a3-4068-9835-50cfd2156350-host-var-lib-kubelet\") pod \"multus-rwwj5\" (UID: \"9a63bf8a-50a3-4068-9835-50cfd2156350\") " pod="openshift-multus/multus-rwwj5" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511219 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511253 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-lib-modules\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511279 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-sys\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511302 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-host-slash\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511365 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511377 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-sys\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511394 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-ovn\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511408 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-lib-modules\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511417 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511423 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-host\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511367 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e0f1aae-624b-4735-ae9b-d27b716a6d9c-host-slash\") pod \"iptables-alerter-5skrq\" (UID: \"0e0f1aae-624b-4735-ae9b-d27b716a6d9c\") " pod="openshift-network-operator/iptables-alerter-5skrq" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.511443 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls podName:cfc1709e-b549-40fc-9471-02c0d106ee9f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:28.01142584 +0000 UTC m=+1.983987011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls") pod "dns-default-9pzjb" (UID: "cfc1709e-b549-40fc-9471-02c0d106ee9f") : secret "dns-default-metrics-tls" not found Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511468 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-ovn\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511506 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-host\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511497 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-modprobe-d\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511558 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-run-netns\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511585 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-run\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511591 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-modprobe-d\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511609 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-cni-bin\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511623 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-run-netns\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511625 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-run\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511634 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511635 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51924e26-cd42-4c49-9307-f041b7ab2d92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511646 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-host-cni-bin\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511659 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-registration-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511685 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511709 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511715 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-registration-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511742 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16e9a2c3-c4e3-4e78-9157-926fae93f8a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j87pt\" (UID: \"16e9a2c3-c4e3-4e78-9157-926fae93f8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j87pt" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511762 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511806 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-var-lib-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.511815 5019 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511830 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511855 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-var-lib-openvswitch\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511858 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysctl-d\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.511874 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:28.011858393 +0000 UTC m=+1.984419574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : secret "router-metrics-certs-default" not found Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:27.511909 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:28.011900121 +0000 UTC m=+1.984461299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : configmap references non-existent config key: service-ca.crt Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511920 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1055486-a6b2-425f-9c20-89c93df7e080-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bqhww\" (UID: \"e1055486-a6b2-425f-9c20-89c93df7e080\") " pod="openshift-multus/multus-additional-cni-plugins-bqhww" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511928 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-systemd\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511955 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7bddb76-6f22-430e-9016-ce19e959dd48-run-systemd\") pod \"ovnkube-node-8ptts\" (UID: \"c7bddb76-6f22-430e-9016-ce19e959dd48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511959 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcd42898-f7cc-4198-851b-db51f7ba110a-dbus\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.511968 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f1e9ac2-4936-4d85-89ab-33cc3ef7a684-etc-sysctl-d\") pod \"tuned-nrx4t\" (UID: \"9f1e9ac2-4936-4d85-89ab-33cc3ef7a684\") " pod="openshift-cluster-node-tuning-operator/tuned-nrx4t" Nov 03 18:44:27.512340 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.512000 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcd42898-f7cc-4198-851b-db51f7ba110a-dbus\") pod \"global-pull-secret-syncer-fn8l9\" (UID: \"dcd42898-f7cc-4198-851b-db51f7ba110a\") " pod="kube-system/global-pull-secret-syncer-fn8l9" Nov 03 18:44:27.514532 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.512985 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls\") pod \"image-registry-78c7cd5678-n2phb\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.514532 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.513947 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls\") pod \"image-registry-67986d9d7f-m6wxv\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.514532 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.514245 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de669e87-278f-48af-bab0-625698973092-metrics-certs\") pod \"network-metrics-daemon-l7j4q\" (UID: \"de669e87-278f-48af-bab0-625698973092\") " pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:27.542803 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.542779 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:27.543792 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.543772 5019 scope.go:122] "RemoveContainer" containerID="f7f6eee515b8b8400434a0f9035f753a42c29660c2a778386f698bfae79d48cc" Nov 03 18:44:27.544703 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.544687 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7j4q" Nov 03 18:44:27.552025 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.551995 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:27.565204 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.562593 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:44:27.728923 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.728893 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4_5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2/kube-storage-version-migrator-operator/0.log" Nov 03 18:44:27.982701 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.982671 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67986d9d7f-m6wxv"] Nov 03 18:44:27.985885 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:27.985858 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18d29b5_b9d5_45fd_a470_b0c664a78aeb.slice/crio-5697bf8b34a4762804bf2d1342181c75dcb299860a284041201890ef30860ee2 WatchSource:0}: Error finding container 5697bf8b34a4762804bf2d1342181c75dcb299860a284041201890ef30860ee2: Status 404 returned error can't find the container with id 5697bf8b34a4762804bf2d1342181c75dcb299860a284041201890ef30860ee2 Nov 03 18:44:27.999461 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:27.999395 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l7j4q"] Nov 03 18:44:28.003440 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:28.003417 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde669e87_278f_48af_bab0_625698973092.slice/crio-6cac3de0319e0a535f8dda782b477c640b2571e2b4b151f28a90845ca9b40c45 WatchSource:0}: Error finding container 6cac3de0319e0a535f8dda782b477c640b2571e2b4b151f28a90845ca9b40c45: Status 404 returned error can't find the container with id 6cac3de0319e0a535f8dda782b477c640b2571e2b4b151f28a90845ca9b40c45 Nov 03 18:44:28.005490 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.005466 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78c7cd5678-n2phb"] Nov 03 18:44:28.016145 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.016123 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:28.016224 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.016152 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:28.016224 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.016184 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:28.016343 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.016224 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:28.016343 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.016251 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:28.016343 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:28.016262 5019 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:44:28.016343 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:28.016285 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.016268182 +0000 UTC m=+2.988829365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : configmap references non-existent config key: service-ca.crt Nov 03 18:44:28.016343 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:28.016320 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.016303301 +0000 UTC m=+2.988864481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : secret "router-metrics-certs-default" not found Nov 03 18:44:28.016343 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:28.016340 5019 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:44:28.016566 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:28.016355 5019 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:44:28.016566 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:28.016386 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert podName:0d94b4ea-05d0-4bbd-bf1f-3a39db705964 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.016374213 +0000 UTC m=+2.988935391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9q8tt" (UID: "0d94b4ea-05d0-4bbd-bf1f-3a39db705964") : secret "networking-console-plugin-cert" not found Nov 03 18:44:28.016566 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:28.016394 5019 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:44:28.016566 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:28.016403 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls podName:51924e26-cd42-4c49-9307-f041b7ab2d92 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.016394075 +0000 UTC m=+2.988955242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w77sj" (UID: "51924e26-cd42-4c49-9307-f041b7ab2d92") : secret "insights-runtime-extractor-tls" not found Nov 03 18:44:28.016566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.016383 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:28.016566 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:28.016423 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls podName:c2709bdc-ce75-411d-9554-39e7c779bf8b nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.016414552 +0000 UTC m=+2.988975717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-txhfg" (UID: "c2709bdc-ce75-411d-9554-39e7c779bf8b") : secret "samples-operator-tls" not found Nov 03 18:44:28.016566 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:28.016437 5019 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:44:28.016566 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:28.016461 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls podName:cfc1709e-b549-40fc-9471-02c0d106ee9f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.016454627 +0000 UTC m=+2.989015786 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls") pod "dns-default-9pzjb" (UID: "cfc1709e-b549-40fc-9471-02c0d106ee9f") : secret "dns-default-metrics-tls" not found Nov 03 18:44:28.016566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.016456 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:28.018528 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.018511 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0725d29-b4bf-4d33-8089-2c568ba41b0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4dlvx\" (UID: \"f0725d29-b4bf-4d33-8089-2c568ba41b0f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:28.032225 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:28.032198 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efe5bf0_abca_490f_b431_a2a72f5bbaee.slice/crio-a32d8fabac4e0a8cd8399077a8fa3046ecee4e08b04266328a846d6914058fb4 WatchSource:0}: Error finding container a32d8fabac4e0a8cd8399077a8fa3046ecee4e08b04266328a846d6914058fb4: Status 404 returned error can't find the container with id a32d8fabac4e0a8cd8399077a8fa3046ecee4e08b04266328a846d6914058fb4 Nov 03 18:44:28.143249 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.143215 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" Nov 03 18:44:28.210393 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.209870 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 18:44:28.210508 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.210397 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/1.log" Nov 03 18:44:28.210508 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.210432 5019 generic.go:356] "Generic (PLEG): container finished" podID="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" containerID="4234d8103edfaf2c6c17dd733d7e737e08499c0d7c5229b6b502996ab205ca1a" exitCode=255 Nov 03 18:44:28.210625 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.210510 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" event={"ID":"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1","Type":"ContainerDied","Data":"4234d8103edfaf2c6c17dd733d7e737e08499c0d7c5229b6b502996ab205ca1a"} Nov 03 18:44:28.210625 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.210564 5019 scope.go:122] "RemoveContainer" containerID="f7f6eee515b8b8400434a0f9035f753a42c29660c2a778386f698bfae79d48cc" Nov 03 18:44:28.211624 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.210987 5019 scope.go:122] "RemoveContainer" containerID="4234d8103edfaf2c6c17dd733d7e737e08499c0d7c5229b6b502996ab205ca1a" Nov 03 18:44:28.211624 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:28.211467 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-48wh4_openshift-console-operator(d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" podUID="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" Nov 03 18:44:28.214467 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.214442 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" event={"ID":"3efe5bf0-abca-490f-b431-a2a72f5bbaee","Type":"ContainerStarted","Data":"79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379"} Nov 03 18:44:28.214552 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.214496 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" event={"ID":"3efe5bf0-abca-490f-b431-a2a72f5bbaee","Type":"ContainerStarted","Data":"a32d8fabac4e0a8cd8399077a8fa3046ecee4e08b04266328a846d6914058fb4"} Nov 03 18:44:28.215765 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.215744 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7j4q" event={"ID":"de669e87-278f-48af-bab0-625698973092","Type":"ContainerStarted","Data":"6cac3de0319e0a535f8dda782b477c640b2571e2b4b151f28a90845ca9b40c45"} Nov 03 18:44:28.217845 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.217818 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" event={"ID":"b18d29b5-b9d5-45fd-a470-b0c664a78aeb","Type":"ContainerStarted","Data":"0d679bd534cbb0c1101adf95a283cc9ae30aa7b845785a866cb2831e5b93fe75"} Nov 03 18:44:28.217939 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.217861 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" event={"ID":"b18d29b5-b9d5-45fd-a470-b0c664a78aeb","Type":"ContainerStarted","Data":"5697bf8b34a4762804bf2d1342181c75dcb299860a284041201890ef30860ee2"} Nov 03 18:44:28.510713 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:28.510691 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx"] Nov 03 18:44:28.525346 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:28.513953 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0725d29_b4bf_4d33_8089_2c568ba41b0f.slice/crio-ba5305345bcd75b2d063692a577c06b4200783fd9095b9591a6cbd4b3260ade6 WatchSource:0}: Error finding container ba5305345bcd75b2d063692a577c06b4200783fd9095b9591a6cbd4b3260ade6: Status 404 returned error can't find the container with id ba5305345bcd75b2d063692a577c06b4200783fd9095b9591a6cbd4b3260ade6 Nov 03 18:44:29.027110 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.027073 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:29.027277 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.027134 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:29.027277 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.027159 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:29.027277 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.027206 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:29.027277 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:29.027243 5019 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:44:29.027277 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.027271 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:29.027561 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:29.027290 5019 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:44:29.027561 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.027301 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:29.027561 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:29.027360 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert podName:0d94b4ea-05d0-4bbd-bf1f-3a39db705964 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:31.027318535 +0000 UTC m=+4.999879699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9q8tt" (UID: "0d94b4ea-05d0-4bbd-bf1f-3a39db705964") : secret "networking-console-plugin-cert" not found Nov 03 18:44:29.027561 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:29.027390 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls podName:51924e26-cd42-4c49-9307-f041b7ab2d92 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:31.027379547 +0000 UTC m=+4.999940710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w77sj" (UID: "51924e26-cd42-4c49-9307-f041b7ab2d92") : secret "insights-runtime-extractor-tls" not found Nov 03 18:44:29.027561 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:29.027415 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:31.027399722 +0000 UTC m=+4.999960889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : configmap references non-existent config key: service-ca.crt Nov 03 18:44:29.027561 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:29.027443 5019 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:44:29.027561 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:29.027477 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls podName:cfc1709e-b549-40fc-9471-02c0d106ee9f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:31.027467224 +0000 UTC m=+5.000028385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls") pod "dns-default-9pzjb" (UID: "cfc1709e-b549-40fc-9471-02c0d106ee9f") : secret "dns-default-metrics-tls" not found Nov 03 18:44:29.030028 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.030003 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c875c2-ea97-435e-821a-067bc0d5c58e-metrics-certs\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:29.030151 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.030009 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2709bdc-ce75-411d-9554-39e7c779bf8b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-txhfg\" (UID: \"c2709bdc-ce75-411d-9554-39e7c779bf8b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:29.041803 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.041780 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" Nov 03 18:44:29.222977 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.222945 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tqfcb" event={"ID":"ad08468c-b770-47bb-82b9-8c0d4d32f855","Type":"ContainerStarted","Data":"77b378bcb5376c95e60e62e0d716d45b93034ca6c00858d581b6059271ae211c"} Nov 03 18:44:29.224703 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.224679 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jnnvk" event={"ID":"d2ca4100-fa85-49bd-bf74-1339c6116298","Type":"ContainerStarted","Data":"8385377b546203f826b8ae1ecfb6f55a51c42e91d52603d680fe9a7f78078902"} Nov 03 18:44:29.226090 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.226067 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" event={"ID":"f0725d29-b4bf-4d33-8089-2c568ba41b0f","Type":"ContainerStarted","Data":"ba5305345bcd75b2d063692a577c06b4200783fd9095b9591a6cbd4b3260ade6"} Nov 03 18:44:29.227701 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.227679 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 18:44:29.227800 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.227779 5019 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:44:29.227850 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.227838 5019 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:44:29.413618 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.413521 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg"] Nov 03 18:44:29.813515 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:29.813485 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:30.233499 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:30.233468 5019 generic.go:356] "Generic (PLEG): container finished" podID="17a9ac1d-c1e8-49d3-9058-72d10b7f662b" containerID="3ce8261598a334d86ea973874ac1972db91996eef3e8030900cca6ff1f3bbb8a" exitCode=0 Nov 03 18:44:30.233914 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:30.233538 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-plzjk" event={"ID":"17a9ac1d-c1e8-49d3-9058-72d10b7f662b","Type":"ContainerDied","Data":"3ce8261598a334d86ea973874ac1972db91996eef3e8030900cca6ff1f3bbb8a"} Nov 03 18:44:30.233971 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:30.233922 5019 scope.go:122] "RemoveContainer" containerID="3ce8261598a334d86ea973874ac1972db91996eef3e8030900cca6ff1f3bbb8a" Nov 03 18:44:30.236133 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:30.235991 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7j4q" event={"ID":"de669e87-278f-48af-bab0-625698973092","Type":"ContainerStarted","Data":"6bc66c5f4fc022fbcf830b2753431cf83e856fad419116c797aecab0d613266f"} Nov 03 18:44:30.236133 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:30.236046 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7j4q" event={"ID":"de669e87-278f-48af-bab0-625698973092","Type":"ContainerStarted","Data":"a1bebb34403998e7318ce557a0d1aefc8e2ce2039b9d5024d41e5df0107ebf69"} Nov 03 18:44:30.237258 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:30.237232 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" event={"ID":"c2709bdc-ce75-411d-9554-39e7c779bf8b","Type":"ContainerStarted","Data":"0a56e5f8d6bb0ebfa662fb673f2bf0cbc1d0a9ebfe6b879972cbc51fb1e29c23"} Nov 03 18:44:30.565649 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:30.565572 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:30.826585 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:30.826512 5019 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:30.826955 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:30.826938 5019 scope.go:122] "RemoveContainer" containerID="4234d8103edfaf2c6c17dd733d7e737e08499c0d7c5229b6b502996ab205ca1a" Nov 03 18:44:30.827187 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:30.827161 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-48wh4_openshift-console-operator(d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" podUID="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" Nov 03 18:44:31.009294 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:31.009261 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:31.046700 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:31.046672 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:31.046854 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:31.046775 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:31.046854 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:31.046844 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:31.046981 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:31.046886 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:31.047039 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:31.047000 5019 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:44:31.047089 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:31.047047 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:35.047026558 +0000 UTC m=+9.019587726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : configmap references non-existent config key: service-ca.crt Nov 03 18:44:31.047089 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:31.047073 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls podName:51924e26-cd42-4c49-9307-f041b7ab2d92 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:35.04706249 +0000 UTC m=+9.019623656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w77sj" (UID: "51924e26-cd42-4c49-9307-f041b7ab2d92") : secret "insights-runtime-extractor-tls" not found Nov 03 18:44:31.047198 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:31.047143 5019 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:44:31.047198 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:31.047191 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert podName:0d94b4ea-05d0-4bbd-bf1f-3a39db705964 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:35.047171838 +0000 UTC m=+9.019733011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9q8tt" (UID: "0d94b4ea-05d0-4bbd-bf1f-3a39db705964") : secret "networking-console-plugin-cert" not found Nov 03 18:44:31.049369 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:31.049340 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfc1709e-b549-40fc-9471-02c0d106ee9f-metrics-tls\") pod \"dns-default-9pzjb\" (UID: \"cfc1709e-b549-40fc-9471-02c0d106ee9f\") " pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:31.153152 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:31.153127 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:31.180406 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:31.180371 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:31.244880 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:31.244843 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-plzjk" event={"ID":"17a9ac1d-c1e8-49d3-9058-72d10b7f662b","Type":"ContainerStarted","Data":"4c518b4690d539823ccf93ca27cb1af88e042a1963299c243a362654cd327738"} Nov 03 18:44:31.249476 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:31.248429 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4dlvx" event={"ID":"f0725d29-b4bf-4d33-8089-2c568ba41b0f","Type":"ContainerStarted","Data":"5041cf7ee075763956fa84e0835a0da74aea0fe5b48ef493890a24c733a4b395"} Nov 03 18:44:31.535755 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:31.535718 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9pzjb"] Nov 03 18:44:31.539009 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:31.538976 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc1709e_b549_40fc_9471_02c0d106ee9f.slice/crio-a19f6152eed5c578a0b26384d0d7beb44ea535f76f3e1f7a5b64df8819dc4cae WatchSource:0}: Error finding container a19f6152eed5c578a0b26384d0d7beb44ea535f76f3e1f7a5b64df8819dc4cae: Status 404 returned error can't find the container with id a19f6152eed5c578a0b26384d0d7beb44ea535f76f3e1f7a5b64df8819dc4cae Nov 03 18:44:31.863993 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:31.863972 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dwp5g" Nov 03 18:44:32.253681 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:32.253636 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9pzjb" event={"ID":"cfc1709e-b549-40fc-9471-02c0d106ee9f","Type":"ContainerStarted","Data":"a19f6152eed5c578a0b26384d0d7beb44ea535f76f3e1f7a5b64df8819dc4cae"} Nov 03 18:44:32.255617 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:32.255586 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" event={"ID":"c2709bdc-ce75-411d-9554-39e7c779bf8b","Type":"ContainerStarted","Data":"ad8884133f95525701d844086c9c5fe9d253baeb045d593a069a80f76516aaa0"} Nov 03 18:44:32.255741 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:32.255635 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-txhfg" event={"ID":"c2709bdc-ce75-411d-9554-39e7c779bf8b","Type":"ContainerStarted","Data":"22f5e1a6dcf110e0d585e4348479358024e82d3d49c886675564c41f172be71f"} Nov 03 18:44:35.082660 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:35.082589 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:35.082660 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:35.082654 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:35.083157 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:35.082693 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:35.083157 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:35.082836 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle podName:d8c875c2-ea97-435e-821a-067bc0d5c58e nodeName:}" failed. No retries permitted until 2026-11-03 18:44:43.082817364 +0000 UTC m=+17.055378535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle") pod "router-default-8dd798ffc-wswht" (UID: "d8c875c2-ea97-435e-821a-067bc0d5c58e") : configmap references non-existent config key: service-ca.crt Nov 03 18:44:35.084962 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:35.084938 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51924e26-cd42-4c49-9307-f041b7ab2d92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w77sj\" (UID: \"51924e26-cd42-4c49-9307-f041b7ab2d92\") " pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:35.085051 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:35.085032 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d94b4ea-05d0-4bbd-bf1f-3a39db705964-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9q8tt\" (UID: \"0d94b4ea-05d0-4bbd-bf1f-3a39db705964\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:35.267138 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:35.267096 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9pzjb" event={"ID":"cfc1709e-b549-40fc-9471-02c0d106ee9f","Type":"ContainerStarted","Data":"cddc2816a1032a40777ad1211fda05edaed53ea5260537f7081f997e4349ef1c"} Nov 03 18:44:35.267277 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:35.267165 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9pzjb" event={"ID":"cfc1709e-b549-40fc-9471-02c0d106ee9f","Type":"ContainerStarted","Data":"6bd548eaf9625f57340392b7ac8896ce116b64857977cfd757a917b1b75f566f"} Nov 03 18:44:35.267277 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:35.267196 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:35.344763 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:35.344693 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" Nov 03 18:44:35.354735 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:35.354713 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w77sj" Nov 03 18:44:35.709997 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:35.709968 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9q8tt"] Nov 03 18:44:35.713118 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:35.713093 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d94b4ea_05d0_4bbd_bf1f_3a39db705964.slice/crio-e9117df6d529530ebb2d5a6111518d77fee46da4ce33d1f3547297b4d9553f9a WatchSource:0}: Error finding container e9117df6d529530ebb2d5a6111518d77fee46da4ce33d1f3547297b4d9553f9a: Status 404 returned error can't find the container with id e9117df6d529530ebb2d5a6111518d77fee46da4ce33d1f3547297b4d9553f9a Nov 03 18:44:35.759120 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:35.759087 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w77sj"] Nov 03 18:44:35.763546 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:35.763522 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51924e26_cd42_4c49_9307_f041b7ab2d92.slice/crio-5167ebfa120130eaa9fd2486d4a82c214c91470e5bf1cb6b969403030f803c8b WatchSource:0}: Error finding container 5167ebfa120130eaa9fd2486d4a82c214c91470e5bf1cb6b969403030f803c8b: Status 404 returned error can't find the container with id 5167ebfa120130eaa9fd2486d4a82c214c91470e5bf1cb6b969403030f803c8b Nov 03 18:44:36.273697 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:36.273662 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" event={"ID":"0d94b4ea-05d0-4bbd-bf1f-3a39db705964","Type":"ContainerStarted","Data":"e9117df6d529530ebb2d5a6111518d77fee46da4ce33d1f3547297b4d9553f9a"} Nov 03 18:44:36.275239 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:36.275209 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w77sj" event={"ID":"51924e26-cd42-4c49-9307-f041b7ab2d92","Type":"ContainerStarted","Data":"ccf8573235e9c4490f30f9cb3bef1784f0271440aac07d555a560168c8429974"} Nov 03 18:44:36.275414 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:36.275255 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w77sj" event={"ID":"51924e26-cd42-4c49-9307-f041b7ab2d92","Type":"ContainerStarted","Data":"5167ebfa120130eaa9fd2486d4a82c214c91470e5bf1cb6b969403030f803c8b"} Nov 03 18:44:36.829757 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:36.829726 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-179.us-west-2.compute.internal" event="NodeReady" Nov 03 18:44:36.869320 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:36.868990 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:36.869320 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:36.869312 5019 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:44:37.280912 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:37.280876 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w77sj" event={"ID":"51924e26-cd42-4c49-9307-f041b7ab2d92","Type":"ContainerStarted","Data":"3b851249c8b813fbaeca327f7b676d0ca27789c13389893b3f16492dbfa08af0"} Nov 03 18:44:37.370392 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:37.370304 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:37.370783 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:37.370765 5019 scope.go:122] "RemoveContainer" containerID="4234d8103edfaf2c6c17dd733d7e737e08499c0d7c5229b6b502996ab205ca1a" Nov 03 18:44:37.371019 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:37.370997 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-48wh4_openshift-console-operator(d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" podUID="d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1" Nov 03 18:44:38.287099 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:38.287053 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-9q8tt" event={"ID":"0d94b4ea-05d0-4bbd-bf1f-3a39db705964","Type":"ContainerStarted","Data":"3ffaffc4cc84a1abfc5087a4f284cd9479714838d2bd50fcf24026d2823632c0"} Nov 03 18:44:38.287559 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:38.287479 5019 scope.go:122] "RemoveContainer" containerID="4234d8103edfaf2c6c17dd733d7e737e08499c0d7c5229b6b502996ab205ca1a" Nov 03 18:44:39.293391 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:39.293307 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 18:44:39.293792 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:39.293437 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" event={"ID":"d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1","Type":"ContainerStarted","Data":"f8e2152b778df9a750179a558e770e6e4aa19b1a951785fc8eb7a454d3748f22"} Nov 03 18:44:39.293792 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:39.293742 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:39.295999 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:39.295971 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w77sj" event={"ID":"51924e26-cd42-4c49-9307-f041b7ab2d92","Type":"ContainerStarted","Data":"3d80e764513b4eeba84f0746fe543420b1312d4fd76e067b38cab5b80ac7b91b"} Nov 03 18:44:39.394109 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:39.394082 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-48wh4" Nov 03 18:44:41.295283 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:44:41.295243 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:44:43.153758 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:43.153717 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:43.154382 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:43.154361 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8c875c2-ea97-435e-821a-067bc0d5c58e-service-ca-bundle\") pod \"router-default-8dd798ffc-wswht\" (UID: \"d8c875c2-ea97-435e-821a-067bc0d5c58e\") " pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:43.442635 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:43.442610 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:43.464076 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:43.464043 5019 patch_prober.go:29] interesting pod/image-registry-78c7cd5678-n2phb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 18:44:43.464180 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:43.464099 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" podUID="3efe5bf0-abca-490f-b431-a2a72f5bbaee" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:44:43.857094 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:43.856937 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8dd798ffc-wswht"] Nov 03 18:44:43.859721 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:43.859684 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c875c2_ea97_435e_821a_067bc0d5c58e.slice/crio-4c99ffe858d139b0567009907907c404cefd2a05973e9b71c489088848af9221 WatchSource:0}: Error finding container 4c99ffe858d139b0567009907907c404cefd2a05973e9b71c489088848af9221: Status 404 returned error can't find the container with id 4c99ffe858d139b0567009907907c404cefd2a05973e9b71c489088848af9221 Nov 03 18:44:44.315926 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:44.315892 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8dd798ffc-wswht" event={"ID":"d8c875c2-ea97-435e-821a-067bc0d5c58e","Type":"ContainerStarted","Data":"ba8f13e1a614b9dcdeaf8a5cb93d8582ff3674f43ae404d74958d727efd2a0b7"} Nov 03 18:44:44.316314 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:44.315944 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8dd798ffc-wswht" event={"ID":"d8c875c2-ea97-435e-821a-067bc0d5c58e","Type":"ContainerStarted","Data":"4c99ffe858d139b0567009907907c404cefd2a05973e9b71c489088848af9221"} Nov 03 18:44:44.443565 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:44.443530 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:44.445870 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:44.445847 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:45.114310 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.114280 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8"] Nov 03 18:44:45.117640 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.117580 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-4rzzr"] Nov 03 18:44:45.117762 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.117703 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8" Nov 03 18:44:45.120513 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.120493 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-4rzzr" Nov 03 18:44:45.161447 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.161419 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67986d9d7f-m6wxv"] Nov 03 18:44:45.165533 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.165510 5019 patch_prober.go:29] interesting pod/image-registry-67986d9d7f-m6wxv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 18:44:45.165650 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.165554 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" podUID="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:44:45.173418 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.173207 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/357d8534-0165-47dc-98ba-9e56da679555-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9njx8\" (UID: \"357d8534-0165-47dc-98ba-9e56da679555\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8" Nov 03 18:44:45.173418 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.173318 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm66c\" (UniqueName: \"kubernetes.io/projected/13c726e0-e525-4127-83df-14b8acb8e1f1-kube-api-access-vm66c\") pod \"downloads-6dd789c8b7-4rzzr\" (UID: \"13c726e0-e525-4127-83df-14b8acb8e1f1\") " pod="openshift-console/downloads-6dd789c8b7-4rzzr" Nov 03 18:44:45.173589 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.173453 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13c726e0-e525-4127-83df-14b8acb8e1f1-tmp\") pod \"downloads-6dd789c8b7-4rzzr\" (UID: \"13c726e0-e525-4127-83df-14b8acb8e1f1\") " pod="openshift-console/downloads-6dd789c8b7-4rzzr" Nov 03 18:44:45.174175 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.174154 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56f955fcf9-srfbb"] Nov 03 18:44:45.177242 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.177227 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.207247 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.207226 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-dxdm2\"" Nov 03 18:44:45.207359 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.207253 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 18:44:45.207359 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.207260 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 18:44:45.207359 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.207263 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 18:44:45.207458 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.207406 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-r26qs\"" Nov 03 18:44:45.236904 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.236875 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-4rzzr"] Nov 03 18:44:45.274108 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.274076 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1835225-4d08-465a-b142-a3b678ba1d3f-ca-trust-extracted\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.274229 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.274139 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1835225-4d08-465a-b142-a3b678ba1d3f-trusted-ca\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.274290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.274271 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1835225-4d08-465a-b142-a3b678ba1d3f-image-registry-private-configuration\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.274368 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.274312 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1835225-4d08-465a-b142-a3b678ba1d3f-registry-tls\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.274436 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.274418 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13c726e0-e525-4127-83df-14b8acb8e1f1-tmp\") pod \"downloads-6dd789c8b7-4rzzr\" (UID: \"13c726e0-e525-4127-83df-14b8acb8e1f1\") " pod="openshift-console/downloads-6dd789c8b7-4rzzr" Nov 03 18:44:45.274484 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.274450 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1835225-4d08-465a-b142-a3b678ba1d3f-bound-sa-token\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.274534 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.274485 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/357d8534-0165-47dc-98ba-9e56da679555-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9njx8\" (UID: \"357d8534-0165-47dc-98ba-9e56da679555\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8" Nov 03 18:44:45.274534 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.274526 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlbq\" (UniqueName: \"kubernetes.io/projected/c1835225-4d08-465a-b142-a3b678ba1d3f-kube-api-access-9hlbq\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.274622 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.274577 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1835225-4d08-465a-b142-a3b678ba1d3f-registry-certificates\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.274622 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.274606 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm66c\" (UniqueName: \"kubernetes.io/projected/13c726e0-e525-4127-83df-14b8acb8e1f1-kube-api-access-vm66c\") pod \"downloads-6dd789c8b7-4rzzr\" (UID: \"13c726e0-e525-4127-83df-14b8acb8e1f1\") " pod="openshift-console/downloads-6dd789c8b7-4rzzr" Nov 03 18:44:45.274698 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.274633 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1835225-4d08-465a-b142-a3b678ba1d3f-installation-pull-secrets\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.274808 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.274782 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13c726e0-e525-4127-83df-14b8acb8e1f1-tmp\") pod \"downloads-6dd789c8b7-4rzzr\" (UID: \"13c726e0-e525-4127-83df-14b8acb8e1f1\") " pod="openshift-console/downloads-6dd789c8b7-4rzzr" Nov 03 18:44:45.277223 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.277202 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/357d8534-0165-47dc-98ba-9e56da679555-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9njx8\" (UID: \"357d8534-0165-47dc-98ba-9e56da679555\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8" Nov 03 18:44:45.277786 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.277768 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9pzjb" Nov 03 18:44:45.295965 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.295943 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56f955fcf9-srfbb"] Nov 03 18:44:45.319463 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.319427 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:45.320545 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.320526 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8dd798ffc-wswht" Nov 03 18:44:45.332956 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.332934 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8"] Nov 03 18:44:45.344947 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.344926 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm66c\" (UniqueName: \"kubernetes.io/projected/13c726e0-e525-4127-83df-14b8acb8e1f1-kube-api-access-vm66c\") pod \"downloads-6dd789c8b7-4rzzr\" (UID: \"13c726e0-e525-4127-83df-14b8acb8e1f1\") " pod="openshift-console/downloads-6dd789c8b7-4rzzr" Nov 03 18:44:45.375776 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.375712 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1835225-4d08-465a-b142-a3b678ba1d3f-registry-certificates\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.375776 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.375751 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1835225-4d08-465a-b142-a3b678ba1d3f-installation-pull-secrets\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.375940 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.375791 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1835225-4d08-465a-b142-a3b678ba1d3f-ca-trust-extracted\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.375940 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.375835 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1835225-4d08-465a-b142-a3b678ba1d3f-trusted-ca\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.376056 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.375940 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1835225-4d08-465a-b142-a3b678ba1d3f-image-registry-private-configuration\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.376056 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.375975 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1835225-4d08-465a-b142-a3b678ba1d3f-registry-tls\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.376056 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.376022 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1835225-4d08-465a-b142-a3b678ba1d3f-bound-sa-token\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.376197 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.376055 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlbq\" (UniqueName: \"kubernetes.io/projected/c1835225-4d08-465a-b142-a3b678ba1d3f-kube-api-access-9hlbq\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.376392 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.376372 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1835225-4d08-465a-b142-a3b678ba1d3f-ca-trust-extracted\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.376677 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.376653 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1835225-4d08-465a-b142-a3b678ba1d3f-registry-certificates\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.376791 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.376771 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1835225-4d08-465a-b142-a3b678ba1d3f-trusted-ca\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.378599 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.378577 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1835225-4d08-465a-b142-a3b678ba1d3f-registry-tls\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.378675 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.378652 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1835225-4d08-465a-b142-a3b678ba1d3f-image-registry-private-configuration\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.378675 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.378659 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1835225-4d08-465a-b142-a3b678ba1d3f-installation-pull-secrets\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.434061 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.434030 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1835225-4d08-465a-b142-a3b678ba1d3f-bound-sa-token\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.434177 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.434160 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlbq\" (UniqueName: \"kubernetes.io/projected/c1835225-4d08-465a-b142-a3b678ba1d3f-kube-api-access-9hlbq\") pod \"image-registry-56f955fcf9-srfbb\" (UID: \"c1835225-4d08-465a-b142-a3b678ba1d3f\") " pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.434924 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.434910 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8" Nov 03 18:44:45.440109 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.440089 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-4rzzr" Nov 03 18:44:45.489870 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.489789 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:45.802712 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.802690 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8"] Nov 03 18:44:45.805221 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:45.805193 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357d8534_0165_47dc_98ba_9e56da679555.slice/crio-ada6844a92fc15a6738b5bbe305b544a26003711ec8a87f19ce97544ebefe592 WatchSource:0}: Error finding container ada6844a92fc15a6738b5bbe305b544a26003711ec8a87f19ce97544ebefe592: Status 404 returned error can't find the container with id ada6844a92fc15a6738b5bbe305b544a26003711ec8a87f19ce97544ebefe592 Nov 03 18:44:45.851405 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.851296 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-4rzzr"] Nov 03 18:44:45.854120 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:45.854094 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c726e0_e525_4127_83df_14b8acb8e1f1.slice/crio-c5d9caa35894a817e74b90b1e74e6dcd35573f30b00aea1dc9e121e5dd34b6a2 WatchSource:0}: Error finding container c5d9caa35894a817e74b90b1e74e6dcd35573f30b00aea1dc9e121e5dd34b6a2: Status 404 returned error can't find the container with id c5d9caa35894a817e74b90b1e74e6dcd35573f30b00aea1dc9e121e5dd34b6a2 Nov 03 18:44:45.903868 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:45.903848 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56f955fcf9-srfbb"] Nov 03 18:44:45.906473 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:45.906447 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1835225_4d08_465a_b142_a3b678ba1d3f.slice/crio-12d775813559ac194b291690af9bede60ade038c5a637df8d955206f32f0b3b3 WatchSource:0}: Error finding container 12d775813559ac194b291690af9bede60ade038c5a637df8d955206f32f0b3b3: Status 404 returned error can't find the container with id 12d775813559ac194b291690af9bede60ade038c5a637df8d955206f32f0b3b3 Nov 03 18:44:46.324269 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:46.324233 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-4rzzr" event={"ID":"13c726e0-e525-4127-83df-14b8acb8e1f1","Type":"ContainerStarted","Data":"c5d9caa35894a817e74b90b1e74e6dcd35573f30b00aea1dc9e121e5dd34b6a2"} Nov 03 18:44:46.325538 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:46.325508 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8" event={"ID":"357d8534-0165-47dc-98ba-9e56da679555","Type":"ContainerStarted","Data":"ada6844a92fc15a6738b5bbe305b544a26003711ec8a87f19ce97544ebefe592"} Nov 03 18:44:46.326970 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:46.326942 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" event={"ID":"c1835225-4d08-465a-b142-a3b678ba1d3f","Type":"ContainerStarted","Data":"7f850a0dcbd37c9d1e460bf6682283f75d43248f904d07d124e05adfe8790b70"} Nov 03 18:44:46.327062 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:46.326986 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" event={"ID":"c1835225-4d08-465a-b142-a3b678ba1d3f","Type":"ContainerStarted","Data":"12d775813559ac194b291690af9bede60ade038c5a637df8d955206f32f0b3b3"} Nov 03 18:44:46.442348 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:46.442282 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" podStartSLOduration=1.44227069 podStartE2EDuration="1.44227069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:46.441739948 +0000 UTC m=+20.414301130" watchObservedRunningTime="2026-11-03 18:44:46.44227069 +0000 UTC m=+20.414831872" Nov 03 18:44:47.331687 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:47.331655 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:44:48.337923 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:48.337850 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8" event={"ID":"357d8534-0165-47dc-98ba-9e56da679555","Type":"ContainerStarted","Data":"9721414ac8650c0e8f36d0c63d01bc1ef4ee532241698da652a875db8eaec930"} Nov 03 18:44:48.448176 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:48.448123 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8" podStartSLOduration=1.917398151 podStartE2EDuration="3.448111466s" totalImagesPullingTime="1.530713315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:48.447726815 +0000 UTC m=+22.420287998" watchObservedRunningTime="2026-11-03 18:44:48.448111466 +0000 UTC m=+22.420672648" Nov 03 18:44:49.343021 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.342988 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8" Nov 03 18:44:49.348583 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.348556 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9njx8" Nov 03 18:44:49.473715 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.473689 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-74d5cbcdbb-wbc8q"] Nov 03 18:44:49.502170 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.502131 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.551701 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.551670 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 18:44:49.552077 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.552045 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-njjxl\"" Nov 03 18:44:49.552205 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.552103 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 18:44:49.552309 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.552293 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 18:44:49.552391 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.552308 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 18:44:49.552391 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.552292 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 18:44:49.571735 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.571713 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d5cbcdbb-wbc8q"] Nov 03 18:44:49.615247 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.615171 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-serving-cert\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.615247 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.615212 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-oauth-serving-cert\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.615247 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.615242 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-service-ca\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.615506 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.615405 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn2fz\" (UniqueName: \"kubernetes.io/projected/d7f875b9-9c73-4486-8823-12c3100ac220-kube-api-access-vn2fz\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.615506 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.615450 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-console-config\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.615506 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.615492 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7f875b9-9c73-4486-8823-12c3100ac220-tmp\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.615637 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.615538 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-oauth-config\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.716672 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.716640 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7f875b9-9c73-4486-8823-12c3100ac220-tmp\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.716855 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.716707 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-oauth-config\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.716855 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.716737 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-serving-cert\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.716855 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.716758 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-oauth-serving-cert\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.716855 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.716778 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-service-ca\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.716855 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.716807 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn2fz\" (UniqueName: \"kubernetes.io/projected/d7f875b9-9c73-4486-8823-12c3100ac220-kube-api-access-vn2fz\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.716855 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.716830 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-console-config\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.717120 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.717091 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7f875b9-9c73-4486-8823-12c3100ac220-tmp\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.717477 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.717454 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-console-config\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.717951 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.717668 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-oauth-serving-cert\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.717951 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.717927 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-service-ca\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.720167 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.720149 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-oauth-config\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.720290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.720268 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-serving-cert\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.776243 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.776209 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn2fz\" (UniqueName: \"kubernetes.io/projected/d7f875b9-9c73-4486-8823-12c3100ac220-kube-api-access-vn2fz\") pod \"console-74d5cbcdbb-wbc8q\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:49.843646 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:49.843614 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:50.234781 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:50.234749 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d5cbcdbb-wbc8q"] Nov 03 18:44:50.267108 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:50.267075 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f875b9_9c73_4486_8823_12c3100ac220.slice/crio-3df15f9ec44a32d04d49a1db5f020796a5ffd0f23267eaa65ca451e5316e7dac WatchSource:0}: Error finding container 3df15f9ec44a32d04d49a1db5f020796a5ffd0f23267eaa65ca451e5316e7dac: Status 404 returned error can't find the container with id 3df15f9ec44a32d04d49a1db5f020796a5ffd0f23267eaa65ca451e5316e7dac Nov 03 18:44:50.349477 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:50.349442 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d5cbcdbb-wbc8q" event={"ID":"d7f875b9-9c73-4486-8823-12c3100ac220","Type":"ContainerStarted","Data":"3df15f9ec44a32d04d49a1db5f020796a5ffd0f23267eaa65ca451e5316e7dac"} Nov 03 18:44:50.571991 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:50.571916 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:44:50.957371 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:50.957334 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw"] Nov 03 18:44:50.985119 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:50.985093 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.019141 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.019116 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw"] Nov 03 18:44:51.034819 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.034793 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 18:44:51.064686 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.064663 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 18:44:51.064806 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.064665 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9ntmz\"" Nov 03 18:44:51.064806 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.064690 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 18:44:51.130758 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.130722 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5ebef9a-b419-487e-92b3-ed9344044974-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tm5fw\" (UID: \"b5ebef9a-b419-487e-92b3-ed9344044974\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.130921 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.130772 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5ebef9a-b419-487e-92b3-ed9344044974-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tm5fw\" (UID: \"b5ebef9a-b419-487e-92b3-ed9344044974\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.130921 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.130896 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpsxm\" (UniqueName: \"kubernetes.io/projected/b5ebef9a-b419-487e-92b3-ed9344044974-kube-api-access-fpsxm\") pod \"prometheus-operator-9cd89b74c-tm5fw\" (UID: \"b5ebef9a-b419-487e-92b3-ed9344044974\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.131035 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.131008 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5ebef9a-b419-487e-92b3-ed9344044974-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tm5fw\" (UID: \"b5ebef9a-b419-487e-92b3-ed9344044974\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.231780 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.231702 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5ebef9a-b419-487e-92b3-ed9344044974-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tm5fw\" (UID: \"b5ebef9a-b419-487e-92b3-ed9344044974\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.231780 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.231764 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5ebef9a-b419-487e-92b3-ed9344044974-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tm5fw\" (UID: \"b5ebef9a-b419-487e-92b3-ed9344044974\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.231969 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.231812 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpsxm\" (UniqueName: \"kubernetes.io/projected/b5ebef9a-b419-487e-92b3-ed9344044974-kube-api-access-fpsxm\") pod \"prometheus-operator-9cd89b74c-tm5fw\" (UID: \"b5ebef9a-b419-487e-92b3-ed9344044974\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.231969 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.231893 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5ebef9a-b419-487e-92b3-ed9344044974-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tm5fw\" (UID: \"b5ebef9a-b419-487e-92b3-ed9344044974\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.232621 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.232597 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5ebef9a-b419-487e-92b3-ed9344044974-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tm5fw\" (UID: \"b5ebef9a-b419-487e-92b3-ed9344044974\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.234413 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.234386 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5ebef9a-b419-487e-92b3-ed9344044974-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tm5fw\" (UID: \"b5ebef9a-b419-487e-92b3-ed9344044974\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.234512 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.234429 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5ebef9a-b419-487e-92b3-ed9344044974-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tm5fw\" (UID: \"b5ebef9a-b419-487e-92b3-ed9344044974\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.290663 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.290635 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpsxm\" (UniqueName: \"kubernetes.io/projected/b5ebef9a-b419-487e-92b3-ed9344044974-kube-api-access-fpsxm\") pod \"prometheus-operator-9cd89b74c-tm5fw\" (UID: \"b5ebef9a-b419-487e-92b3-ed9344044974\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.331079 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.331052 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" Nov 03 18:44:51.724146 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:51.724109 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw"] Nov 03 18:44:51.756400 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:51.756368 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ebef9a_b419_487e_92b3_ed9344044974.slice/crio-8ec78b87ed7a95b71fd9771d0712864f7cc14c14bce7a3f2bbe11277ae8ca6bf WatchSource:0}: Error finding container 8ec78b87ed7a95b71fd9771d0712864f7cc14c14bce7a3f2bbe11277ae8ca6bf: Status 404 returned error can't find the container with id 8ec78b87ed7a95b71fd9771d0712864f7cc14c14bce7a3f2bbe11277ae8ca6bf Nov 03 18:44:52.363666 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:52.363595 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" event={"ID":"b5ebef9a-b419-487e-92b3-ed9344044974","Type":"ContainerStarted","Data":"8ec78b87ed7a95b71fd9771d0712864f7cc14c14bce7a3f2bbe11277ae8ca6bf"} Nov 03 18:44:55.166817 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:55.166742 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:44:55.384500 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:55.384464 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" event={"ID":"b5ebef9a-b419-487e-92b3-ed9344044974","Type":"ContainerStarted","Data":"9382f0b932fc08c6135ca9970c4cc5d84a51e5a9da1e3bbe0dd5027f3b8201bb"} Nov 03 18:44:55.384696 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:55.384534 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" event={"ID":"b5ebef9a-b419-487e-92b3-ed9344044974","Type":"ContainerStarted","Data":"47068d500fcd2cd5b0369ef570c8704e5300b626139f9154525ef45ca1e99732"} Nov 03 18:44:55.386411 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:55.386379 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d5cbcdbb-wbc8q" event={"ID":"d7f875b9-9c73-4486-8823-12c3100ac220","Type":"ContainerStarted","Data":"1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff"} Nov 03 18:44:55.495785 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:55.495729 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tm5fw" podStartSLOduration=2.872313293 podStartE2EDuration="5.495717423s" totalImagesPullingTime="2.62340413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:55.495222573 +0000 UTC m=+29.467783756" watchObservedRunningTime="2026-11-03 18:44:55.495717423 +0000 UTC m=+29.468278604" Nov 03 18:44:55.607604 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:55.606354 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-74d5cbcdbb-wbc8q" podStartSLOduration=2.1465299 podStartE2EDuration="6.606319278s" totalImagesPullingTime="4.459789378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:55.606183009 +0000 UTC m=+29.578744191" watchObservedRunningTime="2026-11-03 18:44:55.606319278 +0000 UTC m=+29.578880460" Nov 03 18:44:56.638285 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:56.638251 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ptts" Nov 03 18:44:58.448954 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.448918 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b6r9c"] Nov 03 18:44:58.452680 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.452658 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.501679 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.501649 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 18:44:58.501824 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.501758 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ltsf8\"" Nov 03 18:44:58.501824 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.501803 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 18:44:58.548654 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.548611 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 18:44:58.605255 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.605222 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f924bab6-aae8-479e-aa28-72bac7cf6b2f-root\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.605434 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.605269 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f924bab6-aae8-479e-aa28-72bac7cf6b2f-metrics-client-ca\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.605434 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.605387 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-tls\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.605526 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.605451 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f924bab6-aae8-479e-aa28-72bac7cf6b2f-sys\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.605526 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.605482 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-textfile\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.605526 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.605498 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9vpg\" (UniqueName: \"kubernetes.io/projected/f924bab6-aae8-479e-aa28-72bac7cf6b2f-kube-api-access-h9vpg\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.605526 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.605525 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-wtmp\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.605681 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.605643 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.605734 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.605679 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-accelerators-collector-config\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.706889 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.706796 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f924bab6-aae8-479e-aa28-72bac7cf6b2f-root\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.706889 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.706856 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f924bab6-aae8-479e-aa28-72bac7cf6b2f-metrics-client-ca\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.706889 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.706884 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-tls\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.707182 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.706904 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f924bab6-aae8-479e-aa28-72bac7cf6b2f-root\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.707182 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.706910 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f924bab6-aae8-479e-aa28-72bac7cf6b2f-sys\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.707182 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.706944 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f924bab6-aae8-479e-aa28-72bac7cf6b2f-sys\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.707182 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.706970 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-textfile\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.707182 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.706988 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9vpg\" (UniqueName: \"kubernetes.io/projected/f924bab6-aae8-479e-aa28-72bac7cf6b2f-kube-api-access-h9vpg\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.707182 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.707010 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-wtmp\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.707182 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.707033 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.707182 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.707052 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-accelerators-collector-config\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.707596 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.707211 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-wtmp\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.707596 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.707451 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-textfile\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.707700 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.707664 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f924bab6-aae8-479e-aa28-72bac7cf6b2f-metrics-client-ca\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.707742 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.707725 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-accelerators-collector-config\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.709735 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.709712 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.709875 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.709848 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f924bab6-aae8-479e-aa28-72bac7cf6b2f-node-exporter-tls\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.765221 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.765185 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9vpg\" (UniqueName: \"kubernetes.io/projected/f924bab6-aae8-479e-aa28-72bac7cf6b2f-kube-api-access-h9vpg\") pod \"node-exporter-b6r9c\" (UID: \"f924bab6-aae8-479e-aa28-72bac7cf6b2f\") " pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.767999 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:58.767973 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b6r9c" Nov 03 18:44:58.781549 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:44:58.781522 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf924bab6_aae8_479e_aa28_72bac7cf6b2f.slice/crio-dff0ed3449f4c967421db6bb94243c65a55e0f9909eefe802a8af581f4055e22 WatchSource:0}: Error finding container dff0ed3449f4c967421db6bb94243c65a55e0f9909eefe802a8af581f4055e22: Status 404 returned error can't find the container with id dff0ed3449f4c967421db6bb94243c65a55e0f9909eefe802a8af581f4055e22 Nov 03 18:44:59.407382 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:59.407346 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b6r9c" event={"ID":"f924bab6-aae8-479e-aa28-72bac7cf6b2f","Type":"ContainerStarted","Data":"dff0ed3449f4c967421db6bb94243c65a55e0f9909eefe802a8af581f4055e22"} Nov 03 18:44:59.817024 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:59.816992 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g2vhp" Nov 03 18:44:59.844838 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:59.844798 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:59.844996 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:59.844849 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:44:59.850347 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:44:59.850270 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:45:00.417924 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:00.417897 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:45:01.419243 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.419201 5019 generic.go:356] "Generic (PLEG): container finished" podID="f924bab6-aae8-479e-aa28-72bac7cf6b2f" containerID="130f58941e29610dfcf45101985176e04c11cf4e9a2cb80970593c3da39af22f" exitCode=0 Nov 03 18:45:01.419819 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.419233 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b6r9c" event={"ID":"f924bab6-aae8-479e-aa28-72bac7cf6b2f","Type":"ContainerDied","Data":"130f58941e29610dfcf45101985176e04c11cf4e9a2cb80970593c3da39af22f"} Nov 03 18:45:01.784884 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.784804 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-54cc49c774-sxmjj"] Nov 03 18:45:01.788480 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.788458 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:01.887755 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.887716 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 18:45:01.887989 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.887970 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cc49c774-sxmjj"] Nov 03 18:45:01.940282 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.940248 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-serving-cert\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:01.940488 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.940296 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-oauth-config\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:01.940488 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.940443 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-console-config\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:01.940617 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.940558 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-oauth-serving-cert\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:01.940617 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.940607 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3afb192-9035-40ad-bae4-020f925182d8-tmp\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:01.940721 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.940642 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-trusted-ca-bundle\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:01.940763 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.940731 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-service-ca\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:01.940805 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:01.940761 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9cph\" (UniqueName: \"kubernetes.io/projected/e3afb192-9035-40ad-bae4-020f925182d8-kube-api-access-q9cph\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.041926 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.041844 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-serving-cert\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.041926 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.041899 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-oauth-config\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.042148 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.041938 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-console-config\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.042148 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.041986 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-oauth-serving-cert\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.042148 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.042029 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3afb192-9035-40ad-bae4-020f925182d8-tmp\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.042148 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.042058 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-trusted-ca-bundle\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.042148 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.042098 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-service-ca\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.042148 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.042125 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9cph\" (UniqueName: \"kubernetes.io/projected/e3afb192-9035-40ad-bae4-020f925182d8-kube-api-access-q9cph\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.043010 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.042977 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-oauth-serving-cert\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.043295 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.043255 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3afb192-9035-40ad-bae4-020f925182d8-tmp\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.043673 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.043648 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-console-config\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.043757 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.043677 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-service-ca\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.043838 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.043814 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-trusted-ca-bundle\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.044946 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.044922 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-oauth-config\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.046843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.046819 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-serving-cert\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.100784 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.100749 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9cph\" (UniqueName: \"kubernetes.io/projected/e3afb192-9035-40ad-bae4-020f925182d8-kube-api-access-q9cph\") pod \"console-54cc49c774-sxmjj\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.101466 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.101441 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:02.427194 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.427156 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b6r9c" event={"ID":"f924bab6-aae8-479e-aa28-72bac7cf6b2f","Type":"ContainerStarted","Data":"8e4be6fad8bc739d2d88ea7cbf8bb2b46e72c2d473919072f0511637a13eeceb"} Nov 03 18:45:02.427698 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.427231 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b6r9c" event={"ID":"f924bab6-aae8-479e-aa28-72bac7cf6b2f","Type":"ContainerStarted","Data":"af03b11065b6ac654cb89d925004a36a6418a78633d7f926477380764c1a12ba"} Nov 03 18:45:02.480067 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.480044 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cc49c774-sxmjj"] Nov 03 18:45:02.482684 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:45:02.482658 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3afb192_9035_40ad_bae4_020f925182d8.slice/crio-e6297bb8e6a9ec9d5d7c84bcf20d1ccbe6fd3ee703727a30e3e3044fa86b36dd WatchSource:0}: Error finding container e6297bb8e6a9ec9d5d7c84bcf20d1ccbe6fd3ee703727a30e3e3044fa86b36dd: Status 404 returned error can't find the container with id e6297bb8e6a9ec9d5d7c84bcf20d1ccbe6fd3ee703727a30e3e3044fa86b36dd Nov 03 18:45:02.545804 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.545748 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b6r9c" podStartSLOduration=2.891179091 podStartE2EDuration="4.545731933s" totalImagesPullingTime="1.576582751s" totalInitContainerRuntime="77.970091ms" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:02.544295785 +0000 UTC m=+36.516856988" watchObservedRunningTime="2026-11-03 18:45:02.545731933 +0000 UTC m=+36.518293107" Nov 03 18:45:02.838988 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:02.838895 5019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:03.433133 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:03.433093 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-4rzzr" event={"ID":"13c726e0-e525-4127-83df-14b8acb8e1f1","Type":"ContainerStarted","Data":"84f8720db0cbe39159e98e4486e37ecb138fabd16e3b4820e423975798299ab7"} Nov 03 18:45:03.433655 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:03.433299 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-4rzzr" Nov 03 18:45:03.435095 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:03.435066 5019 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-4rzzr container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" start-of-body= Nov 03 18:45:03.435225 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:03.435115 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-4rzzr" podUID="13c726e0-e525-4127-83df-14b8acb8e1f1" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" Nov 03 18:45:03.435356 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:03.435311 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cc49c774-sxmjj" event={"ID":"e3afb192-9035-40ad-bae4-020f925182d8","Type":"ContainerStarted","Data":"2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a"} Nov 03 18:45:03.435426 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:03.435379 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cc49c774-sxmjj" event={"ID":"e3afb192-9035-40ad-bae4-020f925182d8","Type":"ContainerStarted","Data":"e6297bb8e6a9ec9d5d7c84bcf20d1ccbe6fd3ee703727a30e3e3044fa86b36dd"} Nov 03 18:45:03.545024 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:03.544958 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-4rzzr" podStartSLOduration=1.070419045 podStartE2EDuration="18.544939142s" totalImagesPullingTime="17.474520097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:03.543421859 +0000 UTC m=+37.515983042" watchObservedRunningTime="2026-11-03 18:45:03.544939142 +0000 UTC m=+37.517500325" Nov 03 18:45:03.658873 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:03.658759 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-54cc49c774-sxmjj" podStartSLOduration=2.658739344 podStartE2EDuration="2.658739344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:03.656820342 +0000 UTC m=+37.629381526" watchObservedRunningTime="2026-11-03 18:45:03.658739344 +0000 UTC m=+37.631300526" Nov 03 18:45:04.446284 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:04.446245 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-4rzzr" Nov 03 18:45:08.343420 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:08.343340 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56f955fcf9-srfbb" Nov 03 18:45:08.560736 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:08.560699 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78c7cd5678-n2phb"] Nov 03 18:45:10.182361 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.182286 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" podUID="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" containerName="registry" containerID="cri-o://0d679bd534cbb0c1101adf95a283cc9ae30aa7b845785a866cb2831e5b93fe75" gracePeriod=30 Nov 03 18:45:10.483445 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.483045 5019 generic.go:356] "Generic (PLEG): container finished" podID="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" containerID="0d679bd534cbb0c1101adf95a283cc9ae30aa7b845785a866cb2831e5b93fe75" exitCode=0 Nov 03 18:45:10.483445 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.483094 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" event={"ID":"b18d29b5-b9d5-45fd-a470-b0c664a78aeb","Type":"ContainerDied","Data":"0d679bd534cbb0c1101adf95a283cc9ae30aa7b845785a866cb2831e5b93fe75"} Nov 03 18:45:10.528873 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.528840 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:45:10.631970 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.631941 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls\") pod \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " Nov 03 18:45:10.632138 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.632021 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-ca-trust-extracted\") pod \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " Nov 03 18:45:10.632138 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.632054 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-trusted-ca\") pod \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " Nov 03 18:45:10.632138 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.632076 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-kube-api-access-hphfz\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-kube-api-access-hphfz\") pod \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " Nov 03 18:45:10.632138 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.632127 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-installation-pull-secrets\") pod \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " Nov 03 18:45:10.632393 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.632170 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-image-registry-private-configuration\") pod \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " Nov 03 18:45:10.632393 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.632206 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-bound-sa-token\") pod \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " Nov 03 18:45:10.632393 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.632247 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-certificates\") pod \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\" (UID: \"b18d29b5-b9d5-45fd-a470-b0c664a78aeb\") " Nov 03 18:45:10.633968 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.633934 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-trusted-ca" pod "b18d29b5-b9d5-45fd-a470-b0c664a78aeb" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:10.634501 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.634474 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-certificates" pod "b18d29b5-b9d5-45fd-a470-b0c664a78aeb" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:10.638623 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.638585 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls" pod "b18d29b5-b9d5-45fd-a470-b0c664a78aeb" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:45:10.638753 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.638693 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-kube-api-access-hphfz" pod "b18d29b5-b9d5-45fd-a470-b0c664a78aeb" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb"). InnerVolumeSpecName "kube-api-access-hphfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:45:10.642044 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.642017 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-ca-trust-extracted" pod "b18d29b5-b9d5-45fd-a470-b0c664a78aeb" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:45:10.653517 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.653486 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-installation-pull-secrets" pod "b18d29b5-b9d5-45fd-a470-b0c664a78aeb" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:45:10.653659 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.653630 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-image-registry-private-configuration" pod "b18d29b5-b9d5-45fd-a470-b0c664a78aeb" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:45:10.653922 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.653895 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-bound-sa-token" pod "b18d29b5-b9d5-45fd-a470-b0c664a78aeb" (UID: "b18d29b5-b9d5-45fd-a470-b0c664a78aeb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:45:10.734116 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.734034 5019 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-ca-trust-extracted\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:10.734116 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.734071 5019 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-trusted-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:10.734116 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.734084 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hphfz\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-kube-api-access-hphfz\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:10.734116 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.734098 5019 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-installation-pull-secrets\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:10.734116 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.734112 5019 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-image-registry-private-configuration\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:10.734421 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.734125 5019 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-bound-sa-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:10.734421 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.734137 5019 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-certificates\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:10.734421 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:10.734149 5019 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b18d29b5-b9d5-45fd-a470-b0c664a78aeb-registry-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:11.302175 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:45:11.302104 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:45:11.490065 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:11.490024 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" event={"ID":"b18d29b5-b9d5-45fd-a470-b0c664a78aeb","Type":"ContainerDied","Data":"5697bf8b34a4762804bf2d1342181c75dcb299860a284041201890ef30860ee2"} Nov 03 18:45:11.490065 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:11.490066 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67986d9d7f-m6wxv" Nov 03 18:45:11.490299 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:11.490093 5019 scope.go:122] "RemoveContainer" containerID="0d679bd534cbb0c1101adf95a283cc9ae30aa7b845785a866cb2831e5b93fe75" Nov 03 18:45:11.697478 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:11.697448 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67986d9d7f-m6wxv"] Nov 03 18:45:11.744273 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:11.744242 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67986d9d7f-m6wxv"] Nov 03 18:45:12.101700 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.101618 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:12.101700 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.101688 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:12.106778 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.106753 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:12.149000 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.148972 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" path="/var/lib/kubelet/pods/b18d29b5-b9d5-45fd-a470-b0c664a78aeb/volumes" Nov 03 18:45:12.262863 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.262830 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54cc49c774-sxmjj"] Nov 03 18:45:12.289628 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.289601 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c955dbdc7-5fvpf"] Nov 03 18:45:12.289922 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.289910 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" containerName="registry" Nov 03 18:45:12.289963 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.289923 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" containerName="registry" Nov 03 18:45:12.290012 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.290003 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18d29b5-b9d5-45fd-a470-b0c664a78aeb" containerName="registry" Nov 03 18:45:12.339743 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.339717 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.347006 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.346981 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2tv\" (UniqueName: \"kubernetes.io/projected/5347f8f8-610c-462b-83b6-f2b0699924f7-kube-api-access-xn2tv\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.347111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.347018 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-oauth-config\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.347111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.347075 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-serving-cert\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.347242 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.347180 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-console-config\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.347242 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.347233 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5347f8f8-610c-462b-83b6-f2b0699924f7-tmp\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.347369 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.347260 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-oauth-serving-cert\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.347369 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.347314 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-trusted-ca-bundle\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.347475 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.347418 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-service-ca\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.352862 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.352806 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c955dbdc7-5fvpf"] Nov 03 18:45:12.448428 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.448391 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-trusted-ca-bundle\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.448615 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.448431 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-service-ca\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.448615 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.448481 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2tv\" (UniqueName: \"kubernetes.io/projected/5347f8f8-610c-462b-83b6-f2b0699924f7-kube-api-access-xn2tv\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.448615 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.448516 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-oauth-config\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.448615 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.448564 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-serving-cert\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.448615 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.448589 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-console-config\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.448878 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.448653 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5347f8f8-610c-462b-83b6-f2b0699924f7-tmp\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.448878 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.448685 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-oauth-serving-cert\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.449296 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.449270 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-console-config\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.449510 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.449418 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-oauth-serving-cert\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.449510 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.449457 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-service-ca\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.449674 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.449619 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5347f8f8-610c-462b-83b6-f2b0699924f7-tmp\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.449792 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.449733 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-trusted-ca-bundle\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.451415 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.451396 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-oauth-config\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.451668 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.451648 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-serving-cert\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.498887 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.498860 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:12.506192 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.506166 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2tv\" (UniqueName: \"kubernetes.io/projected/5347f8f8-610c-462b-83b6-f2b0699924f7-kube-api-access-xn2tv\") pod \"console-6c955dbdc7-5fvpf\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:12.662739 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:12.662649 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:13.058473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:13.058446 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c955dbdc7-5fvpf"] Nov 03 18:45:13.061096 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:45:13.061068 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5347f8f8_610c_462b_83b6_f2b0699924f7.slice/crio-af5bab46d9f3325ed32bb979b413de3f98ae1d2d4de79a7e6e9c44d326ae7686 WatchSource:0}: Error finding container af5bab46d9f3325ed32bb979b413de3f98ae1d2d4de79a7e6e9c44d326ae7686: Status 404 returned error can't find the container with id af5bab46d9f3325ed32bb979b413de3f98ae1d2d4de79a7e6e9c44d326ae7686 Nov 03 18:45:13.500964 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:13.500927 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c955dbdc7-5fvpf" event={"ID":"5347f8f8-610c-462b-83b6-f2b0699924f7","Type":"ContainerStarted","Data":"d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede"} Nov 03 18:45:13.501425 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:13.500982 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c955dbdc7-5fvpf" event={"ID":"5347f8f8-610c-462b-83b6-f2b0699924f7","Type":"ContainerStarted","Data":"af5bab46d9f3325ed32bb979b413de3f98ae1d2d4de79a7e6e9c44d326ae7686"} Nov 03 18:45:13.727164 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:13.727102 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c955dbdc7-5fvpf" podStartSLOduration=1.727090385 podStartE2EDuration="1.727090385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:13.725967139 +0000 UTC m=+47.698528321" watchObservedRunningTime="2026-11-03 18:45:13.727090385 +0000 UTC m=+47.699651567" Nov 03 18:45:16.238490 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.238455 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7nsd9-jcfjp"] Nov 03 18:45:16.239002 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.238667 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" podUID="d03ba26a-59c3-47ce-988a-a4b459714565" containerName="insights-gathering" containerID="cri-o://9f218808178fb1a708d2753dabb989f1656be7c6bd16e9febe0fea4ea6e0b5a7" gracePeriod=30 Nov 03 18:45:16.517522 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.517486 5019 generic.go:356] "Generic (PLEG): container finished" podID="d03ba26a-59c3-47ce-988a-a4b459714565" containerID="9f218808178fb1a708d2753dabb989f1656be7c6bd16e9febe0fea4ea6e0b5a7" exitCode=2 Nov 03 18:45:16.517662 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.517562 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" event={"ID":"d03ba26a-59c3-47ce-988a-a4b459714565","Type":"ContainerDied","Data":"9f218808178fb1a708d2753dabb989f1656be7c6bd16e9febe0fea4ea6e0b5a7"} Nov 03 18:45:16.556727 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.556698 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:45:16.585009 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.584984 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-service-ca-bundle\") pod \"d03ba26a-59c3-47ce-988a-a4b459714565\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " Nov 03 18:45:16.585134 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.585017 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-trusted-ca-bundle\") pod \"d03ba26a-59c3-47ce-988a-a4b459714565\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " Nov 03 18:45:16.585134 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.585043 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03ba26a-59c3-47ce-988a-a4b459714565-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d03ba26a-59c3-47ce-988a-a4b459714565-archives-path\") pod \"d03ba26a-59c3-47ce-988a-a4b459714565\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " Nov 03 18:45:16.585134 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.585067 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03ba26a-59c3-47ce-988a-a4b459714565-kube-api-access-sgkzv\" (UniqueName: \"kubernetes.io/projected/d03ba26a-59c3-47ce-988a-a4b459714565-kube-api-access-sgkzv\") pod \"d03ba26a-59c3-47ce-988a-a4b459714565\" (UID: \"d03ba26a-59c3-47ce-988a-a4b459714565\") " Nov 03 18:45:16.585474 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.585369 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03ba26a-59c3-47ce-988a-a4b459714565-archives-path" pod "d03ba26a-59c3-47ce-988a-a4b459714565" (UID: "d03ba26a-59c3-47ce-988a-a4b459714565"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:45:16.585550 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.585448 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-trusted-ca-bundle" pod "d03ba26a-59c3-47ce-988a-a4b459714565" (UID: "d03ba26a-59c3-47ce-988a-a4b459714565"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:16.585550 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.585484 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-service-ca-bundle" pod "d03ba26a-59c3-47ce-988a-a4b459714565" (UID: "d03ba26a-59c3-47ce-988a-a4b459714565"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:16.587338 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.587303 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03ba26a-59c3-47ce-988a-a4b459714565-kube-api-access-sgkzv" pod "d03ba26a-59c3-47ce-988a-a4b459714565" (UID: "d03ba26a-59c3-47ce-988a-a4b459714565"). InnerVolumeSpecName "kube-api-access-sgkzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:45:16.685891 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.685861 5019 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-service-ca-bundle\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:16.685891 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.685885 5019 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d03ba26a-59c3-47ce-988a-a4b459714565-trusted-ca-bundle\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:16.685891 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.685895 5019 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d03ba26a-59c3-47ce-988a-a4b459714565-archives-path\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:16.686116 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:16.685904 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgkzv\" (UniqueName: \"kubernetes.io/projected/d03ba26a-59c3-47ce-988a-a4b459714565-kube-api-access-sgkzv\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:17.522824 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:17.522791 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" Nov 03 18:45:17.523240 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:17.522824 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7nsd9-jcfjp" event={"ID":"d03ba26a-59c3-47ce-988a-a4b459714565","Type":"ContainerDied","Data":"8c67c8d21f94be880836d9c751957db2c005ea0bcb8f22679bc6a1694d7158a7"} Nov 03 18:45:17.523240 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:17.522883 5019 scope.go:122] "RemoveContainer" containerID="9f218808178fb1a708d2753dabb989f1656be7c6bd16e9febe0fea4ea6e0b5a7" Nov 03 18:45:17.678549 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:17.678516 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7nsd9-jcfjp"] Nov 03 18:45:17.725233 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:17.725187 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-7nsd9-jcfjp"] Nov 03 18:45:18.148870 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:18.148837 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03ba26a-59c3-47ce-988a-a4b459714565" path="/var/lib/kubelet/pods/d03ba26a-59c3-47ce-988a-a4b459714565/volumes" Nov 03 18:45:22.663512 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:22.663481 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:22.664002 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:22.663542 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:22.668179 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:22.668159 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:23.552455 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:23.552425 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:45:23.712252 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:23.712221 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74d5cbcdbb-wbc8q"] Nov 03 18:45:33.588351 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.588290 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" podUID="3efe5bf0-abca-490f-b431-a2a72f5bbaee" containerName="registry" containerID="cri-o://79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379" gracePeriod=30 Nov 03 18:45:33.872946 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.872886 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:45:33.940306 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.940279 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-trusted-ca\") pod \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " Nov 03 18:45:33.940466 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.940318 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3efe5bf0-abca-490f-b431-a2a72f5bbaee-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3efe5bf0-abca-490f-b431-a2a72f5bbaee-ca-trust-extracted\") pod \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " Nov 03 18:45:33.940466 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.940401 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-image-registry-private-configuration\") pod \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " Nov 03 18:45:33.940466 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.940436 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-certificates\") pod \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " Nov 03 18:45:33.940636 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.940475 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-installation-pull-secrets\") pod \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " Nov 03 18:45:33.940636 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.940526 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-kube-api-access-k6dkd\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-kube-api-access-k6dkd\") pod \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " Nov 03 18:45:33.940636 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.940557 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls\") pod \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " Nov 03 18:45:33.940788 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.940635 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-bound-sa-token\") pod \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\" (UID: \"3efe5bf0-abca-490f-b431-a2a72f5bbaee\") " Nov 03 18:45:33.940788 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.940772 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-trusted-ca" pod "3efe5bf0-abca-490f-b431-a2a72f5bbaee" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:33.940922 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.940901 5019 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-trusted-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:33.941062 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.940954 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-certificates" pod "3efe5bf0-abca-490f-b431-a2a72f5bbaee" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:33.943011 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.942982 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-installation-pull-secrets" pod "3efe5bf0-abca-490f-b431-a2a72f5bbaee" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:45:33.943100 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.943062 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-kube-api-access-k6dkd" pod "3efe5bf0-abca-490f-b431-a2a72f5bbaee" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee"). InnerVolumeSpecName "kube-api-access-k6dkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:45:33.943146 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.943122 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-image-registry-private-configuration" pod "3efe5bf0-abca-490f-b431-a2a72f5bbaee" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:45:33.943314 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.943295 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-bound-sa-token" pod "3efe5bf0-abca-490f-b431-a2a72f5bbaee" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:45:33.943398 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.943308 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls" pod "3efe5bf0-abca-490f-b431-a2a72f5bbaee" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:45:33.949725 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:33.949698 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3efe5bf0-abca-490f-b431-a2a72f5bbaee-ca-trust-extracted" pod "3efe5bf0-abca-490f-b431-a2a72f5bbaee" (UID: "3efe5bf0-abca-490f-b431-a2a72f5bbaee"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:45:34.042199 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.042168 5019 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3efe5bf0-abca-490f-b431-a2a72f5bbaee-ca-trust-extracted\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:34.042199 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.042194 5019 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-image-registry-private-configuration\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:34.042199 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.042205 5019 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-certificates\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:34.042434 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.042215 5019 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3efe5bf0-abca-490f-b431-a2a72f5bbaee-installation-pull-secrets\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:34.042434 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.042224 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6dkd\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-kube-api-access-k6dkd\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:34.042434 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.042233 5019 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-registry-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:34.042434 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.042243 5019 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3efe5bf0-abca-490f-b431-a2a72f5bbaee-bound-sa-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:34.600280 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.600248 5019 generic.go:356] "Generic (PLEG): container finished" podID="3efe5bf0-abca-490f-b431-a2a72f5bbaee" containerID="79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379" exitCode=0 Nov 03 18:45:34.600697 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.600295 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" Nov 03 18:45:34.600697 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.600360 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" event={"ID":"3efe5bf0-abca-490f-b431-a2a72f5bbaee","Type":"ContainerDied","Data":"79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379"} Nov 03 18:45:34.600697 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.600409 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78c7cd5678-n2phb" event={"ID":"3efe5bf0-abca-490f-b431-a2a72f5bbaee","Type":"ContainerDied","Data":"a32d8fabac4e0a8cd8399077a8fa3046ecee4e08b04266328a846d6914058fb4"} Nov 03 18:45:34.600697 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.600431 5019 scope.go:122] "RemoveContainer" containerID="79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379" Nov 03 18:45:34.610507 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.610492 5019 scope.go:122] "RemoveContainer" containerID="79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379" Nov 03 18:45:34.610778 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:45:34.610757 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379\": container with ID starting with 79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379 not found: ID does not exist" containerID="79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379" Nov 03 18:45:34.610826 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.610791 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379"} err="failed to get container status \"79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379\": rpc error: code = NotFound desc = could not find container \"79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379\": container with ID starting with 79d5ceffd71a106da25cd721a93433f83187cd48d008039ff5748e97ee860379 not found: ID does not exist" Nov 03 18:45:34.801784 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.801744 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78c7cd5678-n2phb"] Nov 03 18:45:34.848901 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:34.848868 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78c7cd5678-n2phb"] Nov 03 18:45:36.148572 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:36.148540 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3efe5bf0-abca-490f-b431-a2a72f5bbaee" path="/var/lib/kubelet/pods/3efe5bf0-abca-490f-b431-a2a72f5bbaee/volumes" Nov 03 18:45:38.520112 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.520073 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-54cc49c774-sxmjj" podUID="e3afb192-9035-40ad-bae4-020f925182d8" containerName="console" containerID="cri-o://2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a" gracePeriod=15 Nov 03 18:45:38.823897 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.823871 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:38.883461 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.883434 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-oauth-serving-cert\") pod \"e3afb192-9035-40ad-bae4-020f925182d8\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " Nov 03 18:45:38.883635 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.883469 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3afb192-9035-40ad-bae4-020f925182d8-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3afb192-9035-40ad-bae4-020f925182d8-tmp\") pod \"e3afb192-9035-40ad-bae4-020f925182d8\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " Nov 03 18:45:38.883635 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.883493 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3afb192-9035-40ad-bae4-020f925182d8-kube-api-access-q9cph\" (UniqueName: \"kubernetes.io/projected/e3afb192-9035-40ad-bae4-020f925182d8-kube-api-access-q9cph\") pod \"e3afb192-9035-40ad-bae4-020f925182d8\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " Nov 03 18:45:38.883635 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.883522 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-serving-cert\") pod \"e3afb192-9035-40ad-bae4-020f925182d8\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " Nov 03 18:45:38.883635 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.883573 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-console-config\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-console-config\") pod \"e3afb192-9035-40ad-bae4-020f925182d8\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " Nov 03 18:45:38.883851 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.883720 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-trusted-ca-bundle\") pod \"e3afb192-9035-40ad-bae4-020f925182d8\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " Nov 03 18:45:38.883851 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.883773 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3afb192-9035-40ad-bae4-020f925182d8-tmp" pod "e3afb192-9035-40ad-bae4-020f925182d8" (UID: "e3afb192-9035-40ad-bae4-020f925182d8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:45:38.883851 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.883818 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-oauth-config\") pod \"e3afb192-9035-40ad-bae4-020f925182d8\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " Nov 03 18:45:38.883996 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.883855 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-service-ca\") pod \"e3afb192-9035-40ad-bae4-020f925182d8\" (UID: \"e3afb192-9035-40ad-bae4-020f925182d8\") " Nov 03 18:45:38.883996 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.883915 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-oauth-serving-cert" pod "e3afb192-9035-40ad-bae4-020f925182d8" (UID: "e3afb192-9035-40ad-bae4-020f925182d8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:38.883996 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.883969 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-console-config" pod "e3afb192-9035-40ad-bae4-020f925182d8" (UID: "e3afb192-9035-40ad-bae4-020f925182d8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:38.884236 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.884212 5019 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-oauth-serving-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:38.884236 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.884234 5019 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3afb192-9035-40ad-bae4-020f925182d8-tmp\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:38.884454 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.884234 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-trusted-ca-bundle" pod "e3afb192-9035-40ad-bae4-020f925182d8" (UID: "e3afb192-9035-40ad-bae4-020f925182d8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:38.884454 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.884266 5019 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-console-config\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:38.884454 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.884294 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-service-ca" pod "e3afb192-9035-40ad-bae4-020f925182d8" (UID: "e3afb192-9035-40ad-bae4-020f925182d8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:38.885876 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.885855 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-oauth-config" pod "e3afb192-9035-40ad-bae4-020f925182d8" (UID: "e3afb192-9035-40ad-bae4-020f925182d8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:45:38.885971 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.885893 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-serving-cert" pod "e3afb192-9035-40ad-bae4-020f925182d8" (UID: "e3afb192-9035-40ad-bae4-020f925182d8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:45:38.885971 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.885931 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3afb192-9035-40ad-bae4-020f925182d8-kube-api-access-q9cph" pod "e3afb192-9035-40ad-bae4-020f925182d8" (UID: "e3afb192-9035-40ad-bae4-020f925182d8"). InnerVolumeSpecName "kube-api-access-q9cph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:45:38.985677 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.985642 5019 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-trusted-ca-bundle\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:38.985677 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.985672 5019 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-oauth-config\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:38.985677 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.985682 5019 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3afb192-9035-40ad-bae4-020f925182d8-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:38.985892 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.985691 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9cph\" (UniqueName: \"kubernetes.io/projected/e3afb192-9035-40ad-bae4-020f925182d8-kube-api-access-q9cph\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:38.985892 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:38.985701 5019 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3afb192-9035-40ad-bae4-020f925182d8-console-serving-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:39.622068 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:39.622034 5019 generic.go:356] "Generic (PLEG): container finished" podID="e3afb192-9035-40ad-bae4-020f925182d8" containerID="2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a" exitCode=0 Nov 03 18:45:39.622584 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:39.622096 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cc49c774-sxmjj" Nov 03 18:45:39.622584 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:39.622104 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cc49c774-sxmjj" event={"ID":"e3afb192-9035-40ad-bae4-020f925182d8","Type":"ContainerDied","Data":"2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a"} Nov 03 18:45:39.622584 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:39.622169 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cc49c774-sxmjj" event={"ID":"e3afb192-9035-40ad-bae4-020f925182d8","Type":"ContainerDied","Data":"e6297bb8e6a9ec9d5d7c84bcf20d1ccbe6fd3ee703727a30e3e3044fa86b36dd"} Nov 03 18:45:39.622584 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:39.622188 5019 scope.go:122] "RemoveContainer" containerID="2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a" Nov 03 18:45:39.632509 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:39.632490 5019 scope.go:122] "RemoveContainer" containerID="2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a" Nov 03 18:45:39.633089 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:45:39.633059 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a\": container with ID starting with 2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a not found: ID does not exist" containerID="2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a" Nov 03 18:45:39.633180 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:39.633099 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a"} err="failed to get container status \"2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a\": rpc error: code = NotFound desc = could not find container \"2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a\": container with ID starting with 2947dc3f81b556de7a95d9f9f3120b1f02c1fb236f2da9092120f283795e457a not found: ID does not exist" Nov 03 18:45:39.776544 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:39.776511 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54cc49c774-sxmjj"] Nov 03 18:45:39.823636 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:39.823605 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54cc49c774-sxmjj"] Nov 03 18:45:40.148422 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:40.148391 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3afb192-9035-40ad-bae4-020f925182d8" path="/var/lib/kubelet/pods/e3afb192-9035-40ad-bae4-020f925182d8/volumes" Nov 03 18:45:41.294732 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:45:41.294689 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:45:48.732320 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:48.732281 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-74d5cbcdbb-wbc8q" podUID="d7f875b9-9c73-4486-8823-12c3100ac220" containerName="console" containerID="cri-o://1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff" gracePeriod=15 Nov 03 18:45:49.011041 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.010937 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:45:49.177532 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.177498 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-oauth-config\") pod \"d7f875b9-9c73-4486-8823-12c3100ac220\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " Nov 03 18:45:49.177697 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.177573 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f875b9-9c73-4486-8823-12c3100ac220-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7f875b9-9c73-4486-8823-12c3100ac220-tmp\") pod \"d7f875b9-9c73-4486-8823-12c3100ac220\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " Nov 03 18:45:49.177778 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.177743 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-serving-cert\") pod \"d7f875b9-9c73-4486-8823-12c3100ac220\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " Nov 03 18:45:49.177836 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.177804 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f875b9-9c73-4486-8823-12c3100ac220-tmp" pod "d7f875b9-9c73-4486-8823-12c3100ac220" (UID: "d7f875b9-9c73-4486-8823-12c3100ac220"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:45:49.177894 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.177859 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f875b9-9c73-4486-8823-12c3100ac220-kube-api-access-vn2fz\" (UniqueName: \"kubernetes.io/projected/d7f875b9-9c73-4486-8823-12c3100ac220-kube-api-access-vn2fz\") pod \"d7f875b9-9c73-4486-8823-12c3100ac220\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " Nov 03 18:45:49.177951 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.177906 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-console-config\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-console-config\") pod \"d7f875b9-9c73-4486-8823-12c3100ac220\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " Nov 03 18:45:49.178014 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.177963 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-oauth-serving-cert\") pod \"d7f875b9-9c73-4486-8823-12c3100ac220\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " Nov 03 18:45:49.178014 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.178002 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-service-ca\") pod \"d7f875b9-9c73-4486-8823-12c3100ac220\" (UID: \"d7f875b9-9c73-4486-8823-12c3100ac220\") " Nov 03 18:45:49.178248 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.178229 5019 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7f875b9-9c73-4486-8823-12c3100ac220-tmp\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:49.178355 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.178293 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-console-config" pod "d7f875b9-9c73-4486-8823-12c3100ac220" (UID: "d7f875b9-9c73-4486-8823-12c3100ac220"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:49.178529 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.178496 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-service-ca" pod "d7f875b9-9c73-4486-8823-12c3100ac220" (UID: "d7f875b9-9c73-4486-8823-12c3100ac220"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:49.178674 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.178648 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-oauth-serving-cert" pod "d7f875b9-9c73-4486-8823-12c3100ac220" (UID: "d7f875b9-9c73-4486-8823-12c3100ac220"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:49.179925 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.179901 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f875b9-9c73-4486-8823-12c3100ac220-kube-api-access-vn2fz" pod "d7f875b9-9c73-4486-8823-12c3100ac220" (UID: "d7f875b9-9c73-4486-8823-12c3100ac220"). InnerVolumeSpecName "kube-api-access-vn2fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:45:49.180149 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.180129 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-oauth-config" pod "d7f875b9-9c73-4486-8823-12c3100ac220" (UID: "d7f875b9-9c73-4486-8823-12c3100ac220"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:45:49.180215 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.180186 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-serving-cert" pod "d7f875b9-9c73-4486-8823-12c3100ac220" (UID: "d7f875b9-9c73-4486-8823-12c3100ac220"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:45:49.278877 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.278801 5019 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-oauth-serving-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:49.278877 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.278825 5019 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:49.278877 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.278835 5019 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-oauth-config\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:49.278877 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.278849 5019 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7f875b9-9c73-4486-8823-12c3100ac220-console-serving-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:49.278877 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.278864 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn2fz\" (UniqueName: \"kubernetes.io/projected/d7f875b9-9c73-4486-8823-12c3100ac220-kube-api-access-vn2fz\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:49.278877 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.278879 5019 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7f875b9-9c73-4486-8823-12c3100ac220-console-config\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:49.489111 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489070 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-d98dfcf67-txjbd"] Nov 03 18:45:49.489523 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489503 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d03ba26a-59c3-47ce-988a-a4b459714565" containerName="insights-gathering" Nov 03 18:45:49.489523 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489523 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03ba26a-59c3-47ce-988a-a4b459714565" containerName="insights-gathering" Nov 03 18:45:49.489687 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489549 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d7f875b9-9c73-4486-8823-12c3100ac220" containerName="console" Nov 03 18:45:49.489687 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489556 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f875b9-9c73-4486-8823-12c3100ac220" containerName="console" Nov 03 18:45:49.489687 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489568 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3efe5bf0-abca-490f-b431-a2a72f5bbaee" containerName="registry" Nov 03 18:45:49.489687 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489574 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3efe5bf0-abca-490f-b431-a2a72f5bbaee" containerName="registry" Nov 03 18:45:49.489687 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489590 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e3afb192-9035-40ad-bae4-020f925182d8" containerName="console" Nov 03 18:45:49.489687 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489597 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3afb192-9035-40ad-bae4-020f925182d8" containerName="console" Nov 03 18:45:49.489966 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489701 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03ba26a-59c3-47ce-988a-a4b459714565" containerName="insights-gathering" Nov 03 18:45:49.489966 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489718 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f875b9-9c73-4486-8823-12c3100ac220" containerName="console" Nov 03 18:45:49.489966 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489729 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3efe5bf0-abca-490f-b431-a2a72f5bbaee" containerName="registry" Nov 03 18:45:49.489966 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.489740 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3afb192-9035-40ad-bae4-020f925182d8" containerName="console" Nov 03 18:45:49.515586 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.515554 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.595292 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.595218 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d98dfcf67-txjbd"] Nov 03 18:45:49.664922 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.664890 5019 generic.go:356] "Generic (PLEG): container finished" podID="d7f875b9-9c73-4486-8823-12c3100ac220" containerID="1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff" exitCode=0 Nov 03 18:45:49.665104 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.664938 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d5cbcdbb-wbc8q" Nov 03 18:45:49.665104 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.664978 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d5cbcdbb-wbc8q" event={"ID":"d7f875b9-9c73-4486-8823-12c3100ac220","Type":"ContainerDied","Data":"1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff"} Nov 03 18:45:49.665104 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.665025 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d5cbcdbb-wbc8q" event={"ID":"d7f875b9-9c73-4486-8823-12c3100ac220","Type":"ContainerDied","Data":"3df15f9ec44a32d04d49a1db5f020796a5ffd0f23267eaa65ca451e5316e7dac"} Nov 03 18:45:49.665104 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.665044 5019 scope.go:122] "RemoveContainer" containerID="1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff" Nov 03 18:45:49.673527 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.673505 5019 scope.go:122] "RemoveContainer" containerID="1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff" Nov 03 18:45:49.673800 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:45:49.673776 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff\": container with ID starting with 1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff not found: ID does not exist" containerID="1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff" Nov 03 18:45:49.673878 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.673808 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff"} err="failed to get container status \"1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff\": rpc error: code = NotFound desc = could not find container \"1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff\": container with ID starting with 1a121a42ff9d066f5a8a9e798bbbc318ab507a8631ddabaa34b49e2dcd5341ff not found: ID does not exist" Nov 03 18:45:49.682670 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.682649 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzk8x\" (UniqueName: \"kubernetes.io/projected/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-kube-api-access-hzk8x\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.682773 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.682677 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-console-config\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.682773 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.682726 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-tmp\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.682868 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.682846 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-service-ca\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.682910 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.682894 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-console-serving-cert\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.682978 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.682963 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-console-oauth-config\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.683061 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.683050 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-oauth-serving-cert\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.683098 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.683082 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-trusted-ca-bundle\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.783656 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.783628 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzk8x\" (UniqueName: \"kubernetes.io/projected/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-kube-api-access-hzk8x\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.783656 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.783658 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-console-config\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.784137 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.783692 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-tmp\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.784137 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.783717 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-service-ca\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.784137 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.783739 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-console-serving-cert\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.784137 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.783761 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-console-oauth-config\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.784137 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.783793 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-oauth-serving-cert\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.784137 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.783813 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-trusted-ca-bundle\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.784473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.784138 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-tmp\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.784531 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.784509 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-console-config\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.784587 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.784522 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-service-ca\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.784643 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.784619 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-trusted-ca-bundle\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.784717 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.784695 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-oauth-serving-cert\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.786309 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.786290 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-console-serving-cert\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.786381 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.786300 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-console-oauth-config\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:49.818782 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.818755 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74d5cbcdbb-wbc8q"] Nov 03 18:45:49.822176 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.822155 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74d5cbcdbb-wbc8q"] Nov 03 18:45:49.841361 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:49.841319 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzk8x\" (UniqueName: \"kubernetes.io/projected/b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05-kube-api-access-hzk8x\") pod \"console-d98dfcf67-txjbd\" (UID: \"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05\") " pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:50.140562 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:50.140523 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:45:50.147910 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:50.147884 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f875b9-9c73-4486-8823-12c3100ac220" path="/var/lib/kubelet/pods/d7f875b9-9c73-4486-8823-12c3100ac220/volumes" Nov 03 18:45:50.520315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:50.520288 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d98dfcf67-txjbd"] Nov 03 18:45:50.523392 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:45:50.523365 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c73d6f_a8c2_47ce_a32e_f56f2fbbfa05.slice/crio-51f56f7489e5311fc3309d8417a80f5389ced3d9d2e1d74e6c96408afd42e970 WatchSource:0}: Error finding container 51f56f7489e5311fc3309d8417a80f5389ced3d9d2e1d74e6c96408afd42e970: Status 404 returned error can't find the container with id 51f56f7489e5311fc3309d8417a80f5389ced3d9d2e1d74e6c96408afd42e970 Nov 03 18:45:50.670521 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:50.670477 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d98dfcf67-txjbd" event={"ID":"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05","Type":"ContainerStarted","Data":"0acfb3b19012c64e875b9841c6edfde40df9cff63b1243b16be39f276e58f082"} Nov 03 18:45:50.670719 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:50.670544 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d98dfcf67-txjbd" event={"ID":"b8c73d6f-a8c2-47ce-a32e-f56f2fbbfa05","Type":"ContainerStarted","Data":"51f56f7489e5311fc3309d8417a80f5389ced3d9d2e1d74e6c96408afd42e970"} Nov 03 18:45:50.672819 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:50.672792 5019 generic.go:356] "Generic (PLEG): container finished" podID="11e435f9-427b-45ce-971e-f9b184d09b07" containerID="b84f80a963703bc608a6d8725e5221bc670d472e1c8d9cc720bd531f1cb94e43" exitCode=0 Nov 03 18:45:50.672928 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:50.672849 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" event={"ID":"11e435f9-427b-45ce-971e-f9b184d09b07","Type":"ContainerDied","Data":"b84f80a963703bc608a6d8725e5221bc670d472e1c8d9cc720bd531f1cb94e43"} Nov 03 18:45:50.673148 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:50.673136 5019 scope.go:122] "RemoveContainer" containerID="b84f80a963703bc608a6d8725e5221bc670d472e1c8d9cc720bd531f1cb94e43" Nov 03 18:45:50.783197 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:50.783104 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-d98dfcf67-txjbd" podStartSLOduration=1.7830897129999999 podStartE2EDuration="1.783089713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:50.782816488 +0000 UTC m=+84.755377671" watchObservedRunningTime="2026-11-03 18:45:50.783089713 +0000 UTC m=+84.755650895" Nov 03 18:45:51.678777 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:51.678736 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5d722" event={"ID":"11e435f9-427b-45ce-971e-f9b184d09b07","Type":"ContainerStarted","Data":"99d1c9d53e59c00faa73234aafdf3601fe9665605e91b617a0d2aef65dea7d94"} Nov 03 18:45:51.680346 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:51.680301 5019 generic.go:356] "Generic (PLEG): container finished" podID="5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2" containerID="aff2b7547e2e5d179bbfaa5d3f6388af9b1953a0f218acab735d148443186e85" exitCode=0 Nov 03 18:45:51.680463 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:51.680365 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" event={"ID":"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2","Type":"ContainerDied","Data":"aff2b7547e2e5d179bbfaa5d3f6388af9b1953a0f218acab735d148443186e85"} Nov 03 18:45:51.680647 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:51.680633 5019 scope.go:122] "RemoveContainer" containerID="aff2b7547e2e5d179bbfaa5d3f6388af9b1953a0f218acab735d148443186e85" Nov 03 18:45:52.685836 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:52.685803 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7k8p4" event={"ID":"5a09a125-03b8-4d7a-9c7d-f9e107dd5aa2","Type":"ContainerStarted","Data":"d87de0e11bdd4a7bb81f6cc10ad34d54f4d2e04e1ac5d0632d6d5c2bb945fb8f"} Nov 03 18:45:55.700827 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:55.700799 5019 generic.go:356] "Generic (PLEG): container finished" podID="17a9ac1d-c1e8-49d3-9058-72d10b7f662b" containerID="4c518b4690d539823ccf93ca27cb1af88e042a1963299c243a362654cd327738" exitCode=0 Nov 03 18:45:55.701225 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:55.700868 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-plzjk" event={"ID":"17a9ac1d-c1e8-49d3-9058-72d10b7f662b","Type":"ContainerDied","Data":"4c518b4690d539823ccf93ca27cb1af88e042a1963299c243a362654cd327738"} Nov 03 18:45:55.701225 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:55.700920 5019 scope.go:122] "RemoveContainer" containerID="3ce8261598a334d86ea973874ac1972db91996eef3e8030900cca6ff1f3bbb8a" Nov 03 18:45:55.701352 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:45:55.701245 5019 scope.go:122] "RemoveContainer" containerID="4c518b4690d539823ccf93ca27cb1af88e042a1963299c243a362654cd327738" Nov 03 18:45:55.701496 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:45:55.701466 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-plzjk_openshift-insights(17a9ac1d-c1e8-49d3-9058-72d10b7f662b)\"" pod="openshift-insights/insights-operator-7c4986d449-plzjk" podUID="17a9ac1d-c1e8-49d3-9058-72d10b7f662b" Nov 03 18:46:00.141159 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:00.141124 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:46:00.141573 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:00.141181 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:46:00.148243 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:00.148216 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:46:00.727750 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:00.727721 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d98dfcf67-txjbd" Nov 03 18:46:00.888584 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:00.888547 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c955dbdc7-5fvpf"] Nov 03 18:46:07.143537 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:07.143500 5019 scope.go:122] "RemoveContainer" containerID="4c518b4690d539823ccf93ca27cb1af88e042a1963299c243a362654cd327738" Nov 03 18:46:07.753742 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:07.753711 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-plzjk" event={"ID":"17a9ac1d-c1e8-49d3-9058-72d10b7f662b","Type":"ContainerStarted","Data":"763f2450fbacc09854bfa29170b54908c7c948f2478f61877d759570ba55b3b4"} Nov 03 18:46:11.294719 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:46:11.294685 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:46:25.908046 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:25.908000 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c955dbdc7-5fvpf" podUID="5347f8f8-610c-462b-83b6-f2b0699924f7" containerName="console" containerID="cri-o://d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede" gracePeriod=15 Nov 03 18:46:26.175263 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.175241 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:46:26.268591 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.268559 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-oauth-config\") pod \"5347f8f8-610c-462b-83b6-f2b0699924f7\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " Nov 03 18:46:26.268740 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.268654 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-oauth-serving-cert\") pod \"5347f8f8-610c-462b-83b6-f2b0699924f7\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " Nov 03 18:46:26.268740 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.268696 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-console-config\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-console-config\") pod \"5347f8f8-610c-462b-83b6-f2b0699924f7\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " Nov 03 18:46:26.268862 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.268761 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5347f8f8-610c-462b-83b6-f2b0699924f7-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5347f8f8-610c-462b-83b6-f2b0699924f7-tmp\") pod \"5347f8f8-610c-462b-83b6-f2b0699924f7\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " Nov 03 18:46:26.268862 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.268790 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-service-ca\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-service-ca\") pod \"5347f8f8-610c-462b-83b6-f2b0699924f7\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " Nov 03 18:46:26.268862 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.268822 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-trusted-ca-bundle\") pod \"5347f8f8-610c-462b-83b6-f2b0699924f7\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " Nov 03 18:46:26.268862 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.268853 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-serving-cert\") pod \"5347f8f8-610c-462b-83b6-f2b0699924f7\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " Nov 03 18:46:26.269068 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.268883 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5347f8f8-610c-462b-83b6-f2b0699924f7-kube-api-access-xn2tv\" (UniqueName: \"kubernetes.io/projected/5347f8f8-610c-462b-83b6-f2b0699924f7-kube-api-access-xn2tv\") pod \"5347f8f8-610c-462b-83b6-f2b0699924f7\" (UID: \"5347f8f8-610c-462b-83b6-f2b0699924f7\") " Nov 03 18:46:26.269136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.269108 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5347f8f8-610c-462b-83b6-f2b0699924f7-tmp" pod "5347f8f8-610c-462b-83b6-f2b0699924f7" (UID: "5347f8f8-610c-462b-83b6-f2b0699924f7"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:46:26.269193 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.269149 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-oauth-serving-cert" pod "5347f8f8-610c-462b-83b6-f2b0699924f7" (UID: "5347f8f8-610c-462b-83b6-f2b0699924f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:46:26.269245 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.269205 5019 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5347f8f8-610c-462b-83b6-f2b0699924f7-tmp\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:46:26.269294 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.269277 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-service-ca" pod "5347f8f8-610c-462b-83b6-f2b0699924f7" (UID: "5347f8f8-610c-462b-83b6-f2b0699924f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:46:26.269374 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.269315 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-console-config" pod "5347f8f8-610c-462b-83b6-f2b0699924f7" (UID: "5347f8f8-610c-462b-83b6-f2b0699924f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:46:26.269437 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.269419 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-trusted-ca-bundle" pod "5347f8f8-610c-462b-83b6-f2b0699924f7" (UID: "5347f8f8-610c-462b-83b6-f2b0699924f7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:46:26.270731 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.270701 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-oauth-config" pod "5347f8f8-610c-462b-83b6-f2b0699924f7" (UID: "5347f8f8-610c-462b-83b6-f2b0699924f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:46:26.271003 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.270984 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5347f8f8-610c-462b-83b6-f2b0699924f7-kube-api-access-xn2tv" pod "5347f8f8-610c-462b-83b6-f2b0699924f7" (UID: "5347f8f8-610c-462b-83b6-f2b0699924f7"). InnerVolumeSpecName "kube-api-access-xn2tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:46:26.271054 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.271022 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-serving-cert" pod "5347f8f8-610c-462b-83b6-f2b0699924f7" (UID: "5347f8f8-610c-462b-83b6-f2b0699924f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:46:26.370104 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.370068 5019 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-oauth-serving-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:46:26.370104 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.370098 5019 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-console-config\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:46:26.370104 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.370109 5019 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:46:26.370345 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.370118 5019 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5347f8f8-610c-462b-83b6-f2b0699924f7-trusted-ca-bundle\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:46:26.370345 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.370128 5019 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-serving-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:46:26.370345 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.370137 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn2tv\" (UniqueName: \"kubernetes.io/projected/5347f8f8-610c-462b-83b6-f2b0699924f7-kube-api-access-xn2tv\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:46:26.370345 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.370147 5019 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5347f8f8-610c-462b-83b6-f2b0699924f7-console-oauth-config\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:46:26.828026 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.827985 5019 generic.go:356] "Generic (PLEG): container finished" podID="5347f8f8-610c-462b-83b6-f2b0699924f7" containerID="d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede" exitCode=0 Nov 03 18:46:26.828185 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.828037 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c955dbdc7-5fvpf" Nov 03 18:46:26.828185 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.828062 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c955dbdc7-5fvpf" event={"ID":"5347f8f8-610c-462b-83b6-f2b0699924f7","Type":"ContainerDied","Data":"d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede"} Nov 03 18:46:26.828185 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.828110 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c955dbdc7-5fvpf" event={"ID":"5347f8f8-610c-462b-83b6-f2b0699924f7","Type":"ContainerDied","Data":"af5bab46d9f3325ed32bb979b413de3f98ae1d2d4de79a7e6e9c44d326ae7686"} Nov 03 18:46:26.828185 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.828138 5019 scope.go:122] "RemoveContainer" containerID="d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede" Nov 03 18:46:26.836963 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.836946 5019 scope.go:122] "RemoveContainer" containerID="d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede" Nov 03 18:46:26.837199 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:46:26.837179 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede\": container with ID starting with d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede not found: ID does not exist" containerID="d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede" Nov 03 18:46:26.837266 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.837212 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede"} err="failed to get container status \"d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede\": rpc error: code = NotFound desc = could not find container \"d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede\": container with ID starting with d5731011f58eee14a262cc31addadabf54be85de8775e665fbeb32308060cede not found: ID does not exist" Nov 03 18:46:26.985289 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:26.985258 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c955dbdc7-5fvpf"] Nov 03 18:46:27.032108 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:27.032069 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c955dbdc7-5fvpf"] Nov 03 18:46:28.148359 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:46:28.148310 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5347f8f8-610c-462b-83b6-f2b0699924f7" path="/var/lib/kubelet/pods/5347f8f8-610c-462b-83b6-f2b0699924f7/volumes" Nov 03 18:46:41.295529 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:46:41.295450 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:47:11.295523 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:47:11.295479 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:47:41.295400 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:47:41.295363 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:48:11.295905 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:48:11.295828 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:48:34.753622 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.753589 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx"] Nov 03 18:48:34.754043 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.753958 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5347f8f8-610c-462b-83b6-f2b0699924f7" containerName="console" Nov 03 18:48:34.754043 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.753968 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5347f8f8-610c-462b-83b6-f2b0699924f7" containerName="console" Nov 03 18:48:34.754043 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.754020 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5347f8f8-610c-462b-83b6-f2b0699924f7" containerName="console" Nov 03 18:48:34.756891 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.756874 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.794935 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.794906 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.795047 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.794945 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.795047 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.794969 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87jc\" (UniqueName: \"kubernetes.io/projected/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-kube-api-access-l87jc\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.795138 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.795117 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-manifests\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.795181 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.795164 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-metrics-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.795251 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.795236 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-tmp\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.868590 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.868555 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 18:48:34.868590 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.868575 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 18:48:34.868818 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.868684 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 18:48:34.868818 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.868705 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tqcfl\"" Nov 03 18:48:34.868818 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.868734 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 18:48:34.868818 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.868774 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Nov 03 18:48:34.895798 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.895771 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-manifests\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.895956 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.895824 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-metrics-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.895956 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.895860 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-tmp\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.895956 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.895895 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.895956 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.895946 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.896215 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.895987 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87jc\" (UniqueName: \"kubernetes.io/projected/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-kube-api-access-l87jc\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.896274 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.896230 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-manifests\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.898276 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.898249 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-tmp\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.898470 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.898453 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-metrics-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.898771 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.898751 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:34.898810 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:34.898765 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:35.000039 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:35.000006 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87jc\" (UniqueName: \"kubernetes.io/projected/21aa73e8-9d1f-4b5f-a130-6bbab3ee085e-kube-api-access-l87jc\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-pkhrx\" (UID: \"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:35.024423 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:35.024344 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx"] Nov 03 18:48:35.072023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:35.071993 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:35.545297 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:35.545271 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx"] Nov 03 18:48:35.547500 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:48:35.547467 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21aa73e8_9d1f_4b5f_a130_6bbab3ee085e.slice/crio-143b9ac343917f874e30e407e33c706ed8bad5ae5e232d4a085312dedd434044 WatchSource:0}: Error finding container 143b9ac343917f874e30e407e33c706ed8bad5ae5e232d4a085312dedd434044: Status 404 returned error can't find the container with id 143b9ac343917f874e30e407e33c706ed8bad5ae5e232d4a085312dedd434044 Nov 03 18:48:36.334562 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:36.334526 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" event={"ID":"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e","Type":"ContainerStarted","Data":"143b9ac343917f874e30e407e33c706ed8bad5ae5e232d4a085312dedd434044"} Nov 03 18:48:41.298275 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:48:41.298232 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:48:43.367276 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:43.367234 5019 generic.go:356] "Generic (PLEG): container finished" podID="21aa73e8-9d1f-4b5f-a130-6bbab3ee085e" containerID="306f78861b3c94e095651659f1f57b8ecb3a05281d2f0febe8d8c7cff03d978d" exitCode=0 Nov 03 18:48:43.367656 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:43.367322 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" event={"ID":"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e","Type":"ContainerDied","Data":"306f78861b3c94e095651659f1f57b8ecb3a05281d2f0febe8d8c7cff03d978d"} Nov 03 18:48:45.379318 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:45.379277 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" event={"ID":"21aa73e8-9d1f-4b5f-a130-6bbab3ee085e","Type":"ContainerStarted","Data":"137dfd4334ccf09204be42e7da9b91eacc5eaa7f3c8a57def29c8b356f71e621"} Nov 03 18:48:45.379757 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:45.379465 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:48:45.649675 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:45.649561 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" podStartSLOduration=3.475796845 podStartE2EDuration="11.649546719s" totalImagesPullingTime="7.745215849s" totalInitContainerRuntime="428.534025ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:48:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:48:45.602192749 +0000 UTC m=+259.574753942" watchObservedRunningTime="2026-11-03 18:48:45.649546719 +0000 UTC m=+259.622107900" Nov 03 18:48:56.385238 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:48:56.385204 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-pkhrx" Nov 03 18:49:01.862445 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:01.862407 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5"] Nov 03 18:49:01.871352 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:01.871313 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" Nov 03 18:49:01.930375 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:01.930313 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwm5w\" (UniqueName: \"kubernetes.io/projected/58353d70-ebc7-4ecc-b429-ff9acdecb687-kube-api-access-zwm5w\") pod \"kube-auth-proxy-86dc56b666-v5sf5\" (UID: \"58353d70-ebc7-4ecc-b429-ff9acdecb687\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" Nov 03 18:49:01.930555 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:01.930464 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58353d70-ebc7-4ecc-b429-ff9acdecb687-tls-certs\") pod \"kube-auth-proxy-86dc56b666-v5sf5\" (UID: \"58353d70-ebc7-4ecc-b429-ff9acdecb687\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" Nov 03 18:49:01.930622 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:01.930583 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58353d70-ebc7-4ecc-b429-ff9acdecb687-tmp\") pod \"kube-auth-proxy-86dc56b666-v5sf5\" (UID: \"58353d70-ebc7-4ecc-b429-ff9acdecb687\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" Nov 03 18:49:01.968306 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:01.968272 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 18:49:01.968306 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:01.968295 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 18:49:01.968507 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:01.968275 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c9ztg\"" Nov 03 18:49:01.969642 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:01.969622 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5"] Nov 03 18:49:02.031702 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:02.031666 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58353d70-ebc7-4ecc-b429-ff9acdecb687-tmp\") pod \"kube-auth-proxy-86dc56b666-v5sf5\" (UID: \"58353d70-ebc7-4ecc-b429-ff9acdecb687\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" Nov 03 18:49:02.031901 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:02.031724 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwm5w\" (UniqueName: \"kubernetes.io/projected/58353d70-ebc7-4ecc-b429-ff9acdecb687-kube-api-access-zwm5w\") pod \"kube-auth-proxy-86dc56b666-v5sf5\" (UID: \"58353d70-ebc7-4ecc-b429-ff9acdecb687\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" Nov 03 18:49:02.031901 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:02.031758 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58353d70-ebc7-4ecc-b429-ff9acdecb687-tls-certs\") pod \"kube-auth-proxy-86dc56b666-v5sf5\" (UID: \"58353d70-ebc7-4ecc-b429-ff9acdecb687\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" Nov 03 18:49:02.034018 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:02.033992 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58353d70-ebc7-4ecc-b429-ff9acdecb687-tmp\") pod \"kube-auth-proxy-86dc56b666-v5sf5\" (UID: \"58353d70-ebc7-4ecc-b429-ff9acdecb687\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" Nov 03 18:49:02.034137 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:02.034120 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58353d70-ebc7-4ecc-b429-ff9acdecb687-tls-certs\") pod \"kube-auth-proxy-86dc56b666-v5sf5\" (UID: \"58353d70-ebc7-4ecc-b429-ff9acdecb687\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" Nov 03 18:49:02.089832 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:02.089802 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwm5w\" (UniqueName: \"kubernetes.io/projected/58353d70-ebc7-4ecc-b429-ff9acdecb687-kube-api-access-zwm5w\") pod \"kube-auth-proxy-86dc56b666-v5sf5\" (UID: \"58353d70-ebc7-4ecc-b429-ff9acdecb687\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" Nov 03 18:49:02.185797 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:02.185772 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" Nov 03 18:49:02.551468 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:02.551389 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5"] Nov 03 18:49:02.554592 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:49:02.554560 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58353d70_ebc7_4ecc_b429_ff9acdecb687.slice/crio-25dcf0b1bd6f40d69a3ea5ec9971928c96b4ca40de11acd352e321ef2d8ee720 WatchSource:0}: Error finding container 25dcf0b1bd6f40d69a3ea5ec9971928c96b4ca40de11acd352e321ef2d8ee720: Status 404 returned error can't find the container with id 25dcf0b1bd6f40d69a3ea5ec9971928c96b4ca40de11acd352e321ef2d8ee720 Nov 03 18:49:03.454089 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.454055 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" event={"ID":"58353d70-ebc7-4ecc-b429-ff9acdecb687","Type":"ContainerStarted","Data":"25dcf0b1bd6f40d69a3ea5ec9971928c96b4ca40de11acd352e321ef2d8ee720"} Nov 03 18:49:03.502468 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.502439 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc"] Nov 03 18:49:03.506986 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.506964 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.573125 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.573101 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 18:49:03.573278 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.573101 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 18:49:03.573278 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.573235 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 18:49:03.573460 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.573446 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 18:49:03.573508 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.573492 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dqcwl\"" Nov 03 18:49:03.583429 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.583405 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc"] Nov 03 18:49:03.646117 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.646081 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9daedd72-5c5d-4df3-b8e2-321686766ba3-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.646297 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.646132 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.646297 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.646215 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.646297 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.646277 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmq79\" (UniqueName: \"kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-kube-api-access-dmq79\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.646491 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.646375 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.646491 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.646416 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.646491 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.646441 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.747741 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.747479 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9daedd72-5c5d-4df3-b8e2-321686766ba3-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.747741 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.747531 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.747741 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.747558 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.748027 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.747932 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmq79\" (UniqueName: \"kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-kube-api-access-dmq79\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.748027 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.747991 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.749254 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.748173 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.749254 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.748221 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.749254 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.749188 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.750836 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.750808 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.751180 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.751157 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.752505 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.752461 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9daedd72-5c5d-4df3-b8e2-321686766ba3-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.752649 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.752627 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.805902 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.805867 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmq79\" (UniqueName: \"kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-kube-api-access-dmq79\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.806059 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.805928 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-87bcc\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:03.819042 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:03.819009 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:04.249098 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:04.249064 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc"] Nov 03 18:49:04.252411 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:49:04.252376 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9daedd72_5c5d_4df3_b8e2_321686766ba3.slice/crio-857477b882359e304bc066a18b311eb8009753b7c9af2935c5402244e98d99ee WatchSource:0}: Error finding container 857477b882359e304bc066a18b311eb8009753b7c9af2935c5402244e98d99ee: Status 404 returned error can't find the container with id 857477b882359e304bc066a18b311eb8009753b7c9af2935c5402244e98d99ee Nov 03 18:49:04.458904 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:04.458864 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" event={"ID":"9daedd72-5c5d-4df3-b8e2-321686766ba3","Type":"ContainerStarted","Data":"857477b882359e304bc066a18b311eb8009753b7c9af2935c5402244e98d99ee"} Nov 03 18:49:06.471943 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:06.471906 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" event={"ID":"58353d70-ebc7-4ecc-b429-ff9acdecb687","Type":"ContainerStarted","Data":"70013692a6c76b96aeeefc424cdc801cf0e68a75dfaa75fd7b5b8d4d6e1f40d6"} Nov 03 18:49:06.581369 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:06.581285 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86dc56b666-v5sf5" podStartSLOduration=2.496584776 podStartE2EDuration="5.581266903s" totalImagesPullingTime="3.084682127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:06.580840771 +0000 UTC m=+280.553401952" watchObservedRunningTime="2026-11-03 18:49:06.581266903 +0000 UTC m=+280.553828085" Nov 03 18:49:08.013082 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:08.013042 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:49:08.013418 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:08.013114 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:49:08.482469 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:08.482430 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" event={"ID":"9daedd72-5c5d-4df3-b8e2-321686766ba3","Type":"ContainerStarted","Data":"50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6"} Nov 03 18:49:08.482637 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:08.482504 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:08.688560 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:08.688508 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" podStartSLOduration=1.931057531 podStartE2EDuration="5.68849556s" totalImagesPullingTime="3.757438029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:08.641745972 +0000 UTC m=+282.614307154" watchObservedRunningTime="2026-11-03 18:49:08.68849556 +0000 UTC m=+282.661056742" Nov 03 18:49:09.489219 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:09.489185 5019 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-87bcc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:49:09.489615 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:09.489245 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" podUID="9daedd72-5c5d-4df3-b8e2-321686766ba3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:49:11.295392 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:49:11.295354 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:49:12.488774 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:12.488742 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:49:26.165267 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:26.165243 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 18:49:26.165733 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:49:26.165660 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 18:49:41.295715 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:49:41.295633 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:50:11.300355 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:50:11.300301 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:50:19.603257 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.603223 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2"] Nov 03 18:50:19.606721 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.606700 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.667259 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.667229 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2"] Nov 03 18:50:19.695242 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.695207 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.695242 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.695246 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.695469 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.695264 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.695469 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.695419 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8pwm\" (UniqueName: \"kubernetes.io/projected/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-kube-api-access-d8pwm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.695469 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.695464 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.695601 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.695551 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.695601 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.695582 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.796695 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.796660 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.796889 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.796700 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.796889 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.796720 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.796889 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.796746 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8pwm\" (UniqueName: \"kubernetes.io/projected/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-kube-api-access-d8pwm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.796889 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.796764 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.796889 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.796789 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.796889 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.796814 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.797524 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.797497 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.799211 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.799057 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.799211 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.799154 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.799543 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.799525 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.799604 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.799559 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.854297 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.854231 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8pwm\" (UniqueName: \"kubernetes.io/projected/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-kube-api-access-d8pwm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.854443 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.854293 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fwkh2\" (UID: \"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:19.918116 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:19.918087 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:20.334049 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:20.334024 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2"] Nov 03 18:50:20.338663 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:20.338625 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:50:20.338763 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:20.338695 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:50:20.797979 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:20.797947 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" event={"ID":"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c","Type":"ContainerStarted","Data":"7955a6c02e6f4a6f01ff939e7e1879a10ae87ee8f1b032c6eec966d4d6f9ad03"} Nov 03 18:50:20.798392 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:20.797995 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" event={"ID":"9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c","Type":"ContainerStarted","Data":"229c54572707c9082f9dec1a2b7c5b89f5034e9429978ddf8b04a88f0c674395"} Nov 03 18:50:20.798392 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:20.798038 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:20.957734 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:20.957676 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" podStartSLOduration=1.9576633559999999 podStartE2EDuration="1.957663356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:50:20.957423388 +0000 UTC m=+354.929984569" watchObservedRunningTime="2026-11-03 18:50:20.957663356 +0000 UTC m=+354.930224537" Nov 03 18:50:21.803555 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:21.803524 5019 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fwkh2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:50:21.803920 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:21.803575 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" podUID="9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:50:24.803323 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:24.803295 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fwkh2" Nov 03 18:50:25.031448 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.031413 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc"] Nov 03 18:50:25.031661 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.031637 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" podUID="9daedd72-5c5d-4df3-b8e2-321686766ba3" containerName="discovery" containerID="cri-o://50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6" gracePeriod=30 Nov 03 18:50:25.294223 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.294196 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:50:25.344577 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.344539 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-kubeconfig\") pod \"9daedd72-5c5d-4df3-b8e2-321686766ba3\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " Nov 03 18:50:25.344771 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.344586 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-kube-api-access-dmq79\" (UniqueName: \"kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-kube-api-access-dmq79\") pod \"9daedd72-5c5d-4df3-b8e2-321686766ba3\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " Nov 03 18:50:25.344771 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.344607 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-dns-cert\") pod \"9daedd72-5c5d-4df3-b8e2-321686766ba3\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " Nov 03 18:50:25.344771 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.344680 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-ca-configmap\") pod \"9daedd72-5c5d-4df3-b8e2-321686766ba3\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " Nov 03 18:50:25.344771 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.344735 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9daedd72-5c5d-4df3-b8e2-321686766ba3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9daedd72-5c5d-4df3-b8e2-321686766ba3-local-certs\") pod \"9daedd72-5c5d-4df3-b8e2-321686766ba3\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " Nov 03 18:50:25.344771 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.344764 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-token\" (UniqueName: \"kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-token\") pod \"9daedd72-5c5d-4df3-b8e2-321686766ba3\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " Nov 03 18:50:25.345041 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.344806 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-cacerts\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-cacerts\") pod \"9daedd72-5c5d-4df3-b8e2-321686766ba3\" (UID: \"9daedd72-5c5d-4df3-b8e2-321686766ba3\") " Nov 03 18:50:25.345115 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.345085 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-ca-configmap" pod "9daedd72-5c5d-4df3-b8e2-321686766ba3" (UID: "9daedd72-5c5d-4df3-b8e2-321686766ba3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:50:25.345219 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.345186 5019 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-ca-configmap\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:50:25.347236 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.347209 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-dns-cert" pod "9daedd72-5c5d-4df3-b8e2-321686766ba3" (UID: "9daedd72-5c5d-4df3-b8e2-321686766ba3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:50:25.347480 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.347444 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-kube-api-access-dmq79" pod "9daedd72-5c5d-4df3-b8e2-321686766ba3" (UID: "9daedd72-5c5d-4df3-b8e2-321686766ba3"). InnerVolumeSpecName "kube-api-access-dmq79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:50:25.347566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.347532 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9daedd72-5c5d-4df3-b8e2-321686766ba3-local-certs" pod "9daedd72-5c5d-4df3-b8e2-321686766ba3" (UID: "9daedd72-5c5d-4df3-b8e2-321686766ba3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:50:25.347566 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.347530 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-kubeconfig" pod "9daedd72-5c5d-4df3-b8e2-321686766ba3" (UID: "9daedd72-5c5d-4df3-b8e2-321686766ba3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:50:25.347699 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.347562 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-cacerts" pod "9daedd72-5c5d-4df3-b8e2-321686766ba3" (UID: "9daedd72-5c5d-4df3-b8e2-321686766ba3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:50:25.347699 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.347659 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-token" pod "9daedd72-5c5d-4df3-b8e2-321686766ba3" (UID: "9daedd72-5c5d-4df3-b8e2-321686766ba3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:50:25.446051 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.446024 5019 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-kubeconfig\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:50:25.446051 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.446049 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmq79\" (UniqueName: \"kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-kube-api-access-dmq79\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:50:25.446051 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.446058 5019 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-csr-dns-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:50:25.446272 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.446067 5019 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9daedd72-5c5d-4df3-b8e2-321686766ba3-local-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:50:25.446272 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.446075 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9daedd72-5c5d-4df3-b8e2-321686766ba3-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:50:25.446272 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.446085 5019 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9daedd72-5c5d-4df3-b8e2-321686766ba3-cacerts\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:50:25.820085 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.820004 5019 generic.go:356] "Generic (PLEG): container finished" podID="9daedd72-5c5d-4df3-b8e2-321686766ba3" containerID="50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6" exitCode=0 Nov 03 18:50:25.820085 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.820061 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" Nov 03 18:50:25.820529 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.820092 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" event={"ID":"9daedd72-5c5d-4df3-b8e2-321686766ba3","Type":"ContainerDied","Data":"50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6"} Nov 03 18:50:25.820529 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.820170 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc" event={"ID":"9daedd72-5c5d-4df3-b8e2-321686766ba3","Type":"ContainerDied","Data":"857477b882359e304bc066a18b311eb8009753b7c9af2935c5402244e98d99ee"} Nov 03 18:50:25.820529 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.820192 5019 scope.go:122] "RemoveContainer" containerID="50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6" Nov 03 18:50:25.828688 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.828672 5019 scope.go:122] "RemoveContainer" containerID="50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6" Nov 03 18:50:25.828939 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:50:25.828923 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6\": container with ID starting with 50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6 not found: ID does not exist" containerID="50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6" Nov 03 18:50:25.829002 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.828946 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6"} err="failed to get container status \"50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6\": rpc error: code = NotFound desc = could not find container \"50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6\": container with ID starting with 50866e35bd42e5cc9045f3bab1c3e013581305248751ba3214c778f3622a4fa6 not found: ID does not exist" Nov 03 18:50:25.975126 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:25.975098 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc"] Nov 03 18:50:26.021254 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:26.021215 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-87bcc"] Nov 03 18:50:26.149095 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:50:26.149021 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9daedd72-5c5d-4df3-b8e2-321686766ba3" path="/var/lib/kubelet/pods/9daedd72-5c5d-4df3-b8e2-321686766ba3/volumes" Nov 03 18:50:41.296270 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:50:41.296235 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:51:11.294492 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:51:11.294417 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:51:41.295898 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:51:41.295861 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:52:02.827231 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:02.827195 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd"] Nov 03 18:52:02.827742 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:02.827593 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9daedd72-5c5d-4df3-b8e2-321686766ba3" containerName="discovery" Nov 03 18:52:02.827742 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:02.827605 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9daedd72-5c5d-4df3-b8e2-321686766ba3" containerName="discovery" Nov 03 18:52:02.827742 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:02.827665 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9daedd72-5c5d-4df3-b8e2-321686766ba3" containerName="discovery" Nov 03 18:52:02.830677 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:02.830658 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" Nov 03 18:52:02.887998 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:02.887970 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 18:52:02.934568 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:02.934532 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 18:52:02.934778 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:02.934753 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2k4ss\"" Nov 03 18:52:02.936154 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:02.936134 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd"] Nov 03 18:52:02.955069 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:02.955042 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2zx6\" (UniqueName: \"kubernetes.io/projected/58c3a579-8fbe-4a37-9917-40d0823a0899-kube-api-access-z2zx6\") pod \"limitador-operator-controller-manager-856dd584c5-hk4wd\" (UID: \"58c3a579-8fbe-4a37-9917-40d0823a0899\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" Nov 03 18:52:03.055517 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:03.055481 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2zx6\" (UniqueName: \"kubernetes.io/projected/58c3a579-8fbe-4a37-9917-40d0823a0899-kube-api-access-z2zx6\") pod \"limitador-operator-controller-manager-856dd584c5-hk4wd\" (UID: \"58c3a579-8fbe-4a37-9917-40d0823a0899\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" Nov 03 18:52:03.117299 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:03.117217 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2zx6\" (UniqueName: \"kubernetes.io/projected/58c3a579-8fbe-4a37-9917-40d0823a0899-kube-api-access-z2zx6\") pod \"limitador-operator-controller-manager-856dd584c5-hk4wd\" (UID: \"58c3a579-8fbe-4a37-9917-40d0823a0899\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" Nov 03 18:52:03.143061 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:03.143029 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" Nov 03 18:52:03.503798 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:03.503771 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd"] Nov 03 18:52:03.505723 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:52:03.505691 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c3a579_8fbe_4a37_9917_40d0823a0899.slice/crio-e8c5a0a131aa71c1018398ae05c58f4bfd027a9e9447aaa2d1c32cbb8719517b WatchSource:0}: Error finding container e8c5a0a131aa71c1018398ae05c58f4bfd027a9e9447aaa2d1c32cbb8719517b: Status 404 returned error can't find the container with id e8c5a0a131aa71c1018398ae05c58f4bfd027a9e9447aaa2d1c32cbb8719517b Nov 03 18:52:03.507977 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:03.507959 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:52:04.234048 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:04.233994 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" event={"ID":"58c3a579-8fbe-4a37-9917-40d0823a0899","Type":"ContainerStarted","Data":"e8c5a0a131aa71c1018398ae05c58f4bfd027a9e9447aaa2d1c32cbb8719517b"} Nov 03 18:52:04.819032 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:04.819000 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm"] Nov 03 18:52:04.823547 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:04.823528 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm" Nov 03 18:52:04.872579 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:04.872557 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 18:52:04.919070 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:04.919047 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-gwdms\"" Nov 03 18:52:04.926130 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:04.926110 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm"] Nov 03 18:52:04.971505 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:04.971477 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcn9t\" (UniqueName: \"kubernetes.io/projected/db7c9665-d048-4b53-8412-d91f81854523-kube-api-access-lcn9t\") pod \"dns-operator-controller-manager-65bfd79598-kcfwm\" (UID: \"db7c9665-d048-4b53-8412-d91f81854523\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm" Nov 03 18:52:05.072694 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:05.072607 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcn9t\" (UniqueName: \"kubernetes.io/projected/db7c9665-d048-4b53-8412-d91f81854523-kube-api-access-lcn9t\") pod \"dns-operator-controller-manager-65bfd79598-kcfwm\" (UID: \"db7c9665-d048-4b53-8412-d91f81854523\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm" Nov 03 18:52:05.130903 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:05.130870 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcn9t\" (UniqueName: \"kubernetes.io/projected/db7c9665-d048-4b53-8412-d91f81854523-kube-api-access-lcn9t\") pod \"dns-operator-controller-manager-65bfd79598-kcfwm\" (UID: \"db7c9665-d048-4b53-8412-d91f81854523\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm" Nov 03 18:52:05.136460 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:05.136435 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm" Nov 03 18:52:05.504443 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:05.504393 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm"] Nov 03 18:52:05.507407 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:52:05.507378 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7c9665_d048_4b53_8412_d91f81854523.slice/crio-c607c44079b44c9c9bee4ae37f9eb7ceee79d57ee590f826a4a74f79e333c6bc WatchSource:0}: Error finding container c607c44079b44c9c9bee4ae37f9eb7ceee79d57ee590f826a4a74f79e333c6bc: Status 404 returned error can't find the container with id c607c44079b44c9c9bee4ae37f9eb7ceee79d57ee590f826a4a74f79e333c6bc Nov 03 18:52:06.244374 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:06.244320 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm" event={"ID":"db7c9665-d048-4b53-8412-d91f81854523","Type":"ContainerStarted","Data":"c607c44079b44c9c9bee4ae37f9eb7ceee79d57ee590f826a4a74f79e333c6bc"} Nov 03 18:52:09.260345 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.260231 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" event={"ID":"58c3a579-8fbe-4a37-9917-40d0823a0899","Type":"ContainerStarted","Data":"e389d5c0da5230ca1c74e3ce764bbd66e324f0bd353e6852fefa37ad43644304"} Nov 03 18:52:09.260801 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.260389 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" Nov 03 18:52:09.373725 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.373674 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" podStartSLOduration=1.943229177 podStartE2EDuration="7.373660625s" totalImagesPullingTime="5.430431448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:52:09.372585636 +0000 UTC m=+463.345146821" watchObservedRunningTime="2026-11-03 18:52:09.373660625 +0000 UTC m=+463.346221806" Nov 03 18:52:09.546143 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.546074 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65"] Nov 03 18:52:09.549411 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.549392 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:09.601144 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.601124 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-b2nmw\"" Nov 03 18:52:09.608896 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.608877 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vw59\" (UniqueName: \"kubernetes.io/projected/2c323d6a-00b1-4d77-bae5-d17b69e93685-kube-api-access-9vw59\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vrc65\" (UID: \"2c323d6a-00b1-4d77-bae5-d17b69e93685\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:09.608960 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.608919 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c323d6a-00b1-4d77-bae5-d17b69e93685-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vrc65\" (UID: \"2c323d6a-00b1-4d77-bae5-d17b69e93685\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:09.657174 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.657154 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65"] Nov 03 18:52:09.710178 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.710151 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c323d6a-00b1-4d77-bae5-d17b69e93685-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vrc65\" (UID: \"2c323d6a-00b1-4d77-bae5-d17b69e93685\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:09.710372 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.710320 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vw59\" (UniqueName: \"kubernetes.io/projected/2c323d6a-00b1-4d77-bae5-d17b69e93685-kube-api-access-9vw59\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vrc65\" (UID: \"2c323d6a-00b1-4d77-bae5-d17b69e93685\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:09.710619 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.710603 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c323d6a-00b1-4d77-bae5-d17b69e93685-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vrc65\" (UID: \"2c323d6a-00b1-4d77-bae5-d17b69e93685\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:09.768070 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.768039 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vw59\" (UniqueName: \"kubernetes.io/projected/2c323d6a-00b1-4d77-bae5-d17b69e93685-kube-api-access-9vw59\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vrc65\" (UID: \"2c323d6a-00b1-4d77-bae5-d17b69e93685\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:09.861298 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:09.861230 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:10.265977 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:10.265935 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm" event={"ID":"db7c9665-d048-4b53-8412-d91f81854523","Type":"ContainerStarted","Data":"a6f73b23fe7f565d8e9ba43077cd00c15d63b5bcb297a7c45a8ca7e13e5b0f9e"} Nov 03 18:52:10.266486 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:10.266195 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm" Nov 03 18:52:10.272591 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:10.272545 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65"] Nov 03 18:52:10.274950 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:52:10.274925 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c323d6a_00b1_4d77_bae5_d17b69e93685.slice/crio-71035db3f005ae0143cc93727eb7bd897da66cd0e128318d3609806cc3b2cc6f WatchSource:0}: Error finding container 71035db3f005ae0143cc93727eb7bd897da66cd0e128318d3609806cc3b2cc6f: Status 404 returned error can't find the container with id 71035db3f005ae0143cc93727eb7bd897da66cd0e128318d3609806cc3b2cc6f Nov 03 18:52:10.384397 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:10.384318 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm" podStartSLOduration=2.286611104 podStartE2EDuration="6.384304129s" totalImagesPullingTime="4.097693025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:52:10.383223518 +0000 UTC m=+464.355784701" watchObservedRunningTime="2026-11-03 18:52:10.384304129 +0000 UTC m=+464.356865311" Nov 03 18:52:11.271395 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:11.271352 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" event={"ID":"2c323d6a-00b1-4d77-bae5-d17b69e93685","Type":"ContainerStarted","Data":"71035db3f005ae0143cc93727eb7bd897da66cd0e128318d3609806cc3b2cc6f"} Nov 03 18:52:11.295447 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:11.295412 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:52:18.305514 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:18.305477 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" event={"ID":"2c323d6a-00b1-4d77-bae5-d17b69e93685","Type":"ContainerStarted","Data":"1d31367b6e84f3e3feed8fd5f13d5221d2340ce34993bc6ad6a5625521468263"} Nov 03 18:52:18.305924 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:18.305555 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:18.510987 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:18.510930 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" podStartSLOduration=1.655290607 podStartE2EDuration="9.51091665s" totalImagesPullingTime="7.855626043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:52:18.463642687 +0000 UTC m=+472.436203881" watchObservedRunningTime="2026-11-03 18:52:18.51091665 +0000 UTC m=+472.483477829" Nov 03 18:52:20.268290 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:20.268254 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" Nov 03 18:52:21.273560 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:21.273532 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-kcfwm" Nov 03 18:52:29.311991 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:29.311958 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:30.724249 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:30.724215 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc"] Nov 03 18:52:30.728035 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:30.728015 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:30.794430 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:30.794398 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pznqr\" (UniqueName: \"kubernetes.io/projected/db08f44d-3541-4f28-b6e5-bb5f16e937af-kube-api-access-pznqr\") pod \"kuadrant-operator-controller-manager-6b76784d48-jlgwc\" (UID: \"db08f44d-3541-4f28-b6e5-bb5f16e937af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:30.794782 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:30.794476 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db08f44d-3541-4f28-b6e5-bb5f16e937af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-jlgwc\" (UID: \"db08f44d-3541-4f28-b6e5-bb5f16e937af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:30.830261 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:30.830237 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc"] Nov 03 18:52:30.895960 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:30.895926 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db08f44d-3541-4f28-b6e5-bb5f16e937af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-jlgwc\" (UID: \"db08f44d-3541-4f28-b6e5-bb5f16e937af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:30.896120 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:30.895981 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pznqr\" (UniqueName: \"kubernetes.io/projected/db08f44d-3541-4f28-b6e5-bb5f16e937af-kube-api-access-pznqr\") pod \"kuadrant-operator-controller-manager-6b76784d48-jlgwc\" (UID: \"db08f44d-3541-4f28-b6e5-bb5f16e937af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:30.896348 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:30.896308 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db08f44d-3541-4f28-b6e5-bb5f16e937af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-jlgwc\" (UID: \"db08f44d-3541-4f28-b6e5-bb5f16e937af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:30.953428 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:30.953393 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pznqr\" (UniqueName: \"kubernetes.io/projected/db08f44d-3541-4f28-b6e5-bb5f16e937af-kube-api-access-pznqr\") pod \"kuadrant-operator-controller-manager-6b76784d48-jlgwc\" (UID: \"db08f44d-3541-4f28-b6e5-bb5f16e937af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:31.039981 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:31.039892 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:31.454053 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:31.454029 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc"] Nov 03 18:52:31.457398 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:52:31.457368 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb08f44d_3541_4f28_b6e5_bb5f16e937af.slice/crio-f55235b8b308d3ac762cd85916a8bc553448f01171c89691f69feb9e27856786 WatchSource:0}: Error finding container f55235b8b308d3ac762cd85916a8bc553448f01171c89691f69feb9e27856786: Status 404 returned error can't find the container with id f55235b8b308d3ac762cd85916a8bc553448f01171c89691f69feb9e27856786 Nov 03 18:52:32.369609 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:32.369573 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" event={"ID":"db08f44d-3541-4f28-b6e5-bb5f16e937af","Type":"ContainerStarted","Data":"24d1c1920354ba2b9e2d9eed25227d09f2d37b7ad6cbe9d14117644a26126400"} Nov 03 18:52:32.370008 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:32.369634 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" event={"ID":"db08f44d-3541-4f28-b6e5-bb5f16e937af","Type":"ContainerStarted","Data":"f55235b8b308d3ac762cd85916a8bc553448f01171c89691f69feb9e27856786"} Nov 03 18:52:32.370008 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:32.369655 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:32.580136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:32.580083 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" podStartSLOduration=2.580069292 podStartE2EDuration="2.580069292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:52:32.532098813 +0000 UTC m=+486.504659996" watchObservedRunningTime="2026-11-03 18:52:32.580069292 +0000 UTC m=+486.552630473" Nov 03 18:52:41.295321 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:41.295244 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:52:42.967043 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:42.967007 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65"] Nov 03 18:52:42.967490 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:42.967239 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" podUID="2c323d6a-00b1-4d77-bae5-d17b69e93685" containerName="manager" containerID="cri-o://1d31367b6e84f3e3feed8fd5f13d5221d2340ce34993bc6ad6a5625521468263" gracePeriod=2 Nov 03 18:52:42.968463 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:42.968443 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65"] Nov 03 18:52:43.014670 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.014623 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj"] Nov 03 18:52:43.015079 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.015060 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2c323d6a-00b1-4d77-bae5-d17b69e93685" containerName="manager" Nov 03 18:52:43.015079 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.015078 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c323d6a-00b1-4d77-bae5-d17b69e93685" containerName="manager" Nov 03 18:52:43.015198 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.015167 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c323d6a-00b1-4d77-bae5-d17b69e93685" containerName="manager" Nov 03 18:52:43.023535 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.023511 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc"] Nov 03 18:52:43.023648 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.023561 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc"] Nov 03 18:52:43.023749 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.023714 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" podUID="db08f44d-3541-4f28-b6e5-bb5f16e937af" containerName="manager" containerID="cri-o://24d1c1920354ba2b9e2d9eed25227d09f2d37b7ad6cbe9d14117644a26126400" gracePeriod=2 Nov 03 18:52:43.023869 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.023794 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" Nov 03 18:52:43.058690 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.058632 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-vrc65\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="2c323d6a-00b1-4d77-bae5-d17b69e93685" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:43.060095 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.060076 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws"] Nov 03 18:52:43.060538 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.060521 5019 cpu_manager.go:414] "containerMap: removing container" podUID="db08f44d-3541-4f28-b6e5-bb5f16e937af" containerName="manager" Nov 03 18:52:43.060581 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.060540 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db08f44d-3541-4f28-b6e5-bb5f16e937af" containerName="manager" Nov 03 18:52:43.060627 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.060618 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="db08f44d-3541-4f28-b6e5-bb5f16e937af" containerName="manager" Nov 03 18:52:43.063603 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.063582 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd"] Nov 03 18:52:43.063690 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.063629 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd"] Nov 03 18:52:43.063734 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.063697 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:52:43.063805 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.063786 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt"] Nov 03 18:52:43.063886 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.063853 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" podUID="58c3a579-8fbe-4a37-9917-40d0823a0899" containerName="manager" containerID="cri-o://e389d5c0da5230ca1c74e3ce764bbd66e324f0bd353e6852fefa37ad43644304" gracePeriod=2 Nov 03 18:52:43.064239 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.064168 5019 cpu_manager.go:414] "containerMap: removing container" podUID="58c3a579-8fbe-4a37-9917-40d0823a0899" containerName="manager" Nov 03 18:52:43.064239 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.064181 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c3a579-8fbe-4a37-9917-40d0823a0899" containerName="manager" Nov 03 18:52:43.064310 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.064252 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c3a579-8fbe-4a37-9917-40d0823a0899" containerName="manager" Nov 03 18:52:43.067079 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.067061 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt" Nov 03 18:52:43.086604 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.086575 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj"] Nov 03 18:52:43.103185 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.103157 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t29r\" (UniqueName: \"kubernetes.io/projected/87a43aff-7fd6-4364-8db1-f819fd75da6c-kube-api-access-7t29r\") pod \"kuadrant-operator-controller-manager-6b6fb56846-kffzj\" (UID: \"87a43aff-7fd6-4364-8db1-f819fd75da6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" Nov 03 18:52:43.103314 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.103200 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87a43aff-7fd6-4364-8db1-f819fd75da6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-kffzj\" (UID: \"87a43aff-7fd6-4364-8db1-f819fd75da6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" Nov 03 18:52:43.106444 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.106414 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-vrc65\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="2c323d6a-00b1-4d77-bae5-d17b69e93685" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:43.146928 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.146899 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws"] Nov 03 18:52:43.151855 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.151818 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt"] Nov 03 18:52:43.154015 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.153959 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-jlgwc\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="db08f44d-3541-4f28-b6e5-bb5f16e937af" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:43.204196 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.204167 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswsb\" (UniqueName: \"kubernetes.io/projected/735187df-3794-4b77-a6f2-9ee7ea5803b4-kube-api-access-jswsb\") pod \"limitador-operator-controller-manager-856dd584c5-2mzjt\" (UID: \"735187df-3794-4b77-a6f2-9ee7ea5803b4\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt" Nov 03 18:52:43.204315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.204218 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t29r\" (UniqueName: \"kubernetes.io/projected/87a43aff-7fd6-4364-8db1-f819fd75da6c-kube-api-access-7t29r\") pod \"kuadrant-operator-controller-manager-6b6fb56846-kffzj\" (UID: \"87a43aff-7fd6-4364-8db1-f819fd75da6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" Nov 03 18:52:43.204315 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.204284 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpnjx\" (UniqueName: \"kubernetes.io/projected/dec280db-814f-478f-8824-22a20ec14aee-kube-api-access-hpnjx\") pod \"kuadrant-operator-controller-manager-6b76784d48-m4sws\" (UID: \"dec280db-814f-478f-8824-22a20ec14aee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:52:43.204444 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.204342 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87a43aff-7fd6-4364-8db1-f819fd75da6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-kffzj\" (UID: \"87a43aff-7fd6-4364-8db1-f819fd75da6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" Nov 03 18:52:43.204444 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.204423 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dec280db-814f-478f-8824-22a20ec14aee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-m4sws\" (UID: \"dec280db-814f-478f-8824-22a20ec14aee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:52:43.204966 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.204949 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87a43aff-7fd6-4364-8db1-f819fd75da6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-kffzj\" (UID: \"87a43aff-7fd6-4364-8db1-f819fd75da6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" Nov 03 18:52:43.263422 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.263391 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t29r\" (UniqueName: \"kubernetes.io/projected/87a43aff-7fd6-4364-8db1-f819fd75da6c-kube-api-access-7t29r\") pod \"kuadrant-operator-controller-manager-6b6fb56846-kffzj\" (UID: \"87a43aff-7fd6-4364-8db1-f819fd75da6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" Nov 03 18:52:43.305472 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.305441 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswsb\" (UniqueName: \"kubernetes.io/projected/735187df-3794-4b77-a6f2-9ee7ea5803b4-kube-api-access-jswsb\") pod \"limitador-operator-controller-manager-856dd584c5-2mzjt\" (UID: \"735187df-3794-4b77-a6f2-9ee7ea5803b4\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt" Nov 03 18:52:43.305592 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.305512 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpnjx\" (UniqueName: \"kubernetes.io/projected/dec280db-814f-478f-8824-22a20ec14aee-kube-api-access-hpnjx\") pod \"kuadrant-operator-controller-manager-6b76784d48-m4sws\" (UID: \"dec280db-814f-478f-8824-22a20ec14aee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:52:43.305727 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.305703 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dec280db-814f-478f-8824-22a20ec14aee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-m4sws\" (UID: \"dec280db-814f-478f-8824-22a20ec14aee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:52:43.306158 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.306108 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dec280db-814f-478f-8824-22a20ec14aee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-m4sws\" (UID: \"dec280db-814f-478f-8824-22a20ec14aee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:52:43.312944 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.312916 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-hk4wd\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="58c3a579-8fbe-4a37-9917-40d0823a0899" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" Nov 03 18:52:43.316336 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.316306 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:43.346319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.346298 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:43.352045 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.352027 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" Nov 03 18:52:43.368145 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.368121 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswsb\" (UniqueName: \"kubernetes.io/projected/735187df-3794-4b77-a6f2-9ee7ea5803b4-kube-api-access-jswsb\") pod \"limitador-operator-controller-manager-856dd584c5-2mzjt\" (UID: \"735187df-3794-4b77-a6f2-9ee7ea5803b4\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt" Nov 03 18:52:43.368285 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.368267 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpnjx\" (UniqueName: \"kubernetes.io/projected/dec280db-814f-478f-8824-22a20ec14aee-kube-api-access-hpnjx\") pod \"kuadrant-operator-controller-manager-6b76784d48-m4sws\" (UID: \"dec280db-814f-478f-8824-22a20ec14aee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:52:43.406321 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.406299 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c323d6a-00b1-4d77-bae5-d17b69e93685-kube-api-access-9vw59\" (UniqueName: \"kubernetes.io/projected/2c323d6a-00b1-4d77-bae5-d17b69e93685-kube-api-access-9vw59\") pod \"2c323d6a-00b1-4d77-bae5-d17b69e93685\" (UID: \"2c323d6a-00b1-4d77-bae5-d17b69e93685\") " Nov 03 18:52:43.406423 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.406361 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c323d6a-00b1-4d77-bae5-d17b69e93685-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c323d6a-00b1-4d77-bae5-d17b69e93685-extensions-socket-volume\") pod \"2c323d6a-00b1-4d77-bae5-d17b69e93685\" (UID: \"2c323d6a-00b1-4d77-bae5-d17b69e93685\") " Nov 03 18:52:43.406716 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.406687 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c323d6a-00b1-4d77-bae5-d17b69e93685-extensions-socket-volume" pod "2c323d6a-00b1-4d77-bae5-d17b69e93685" (UID: "2c323d6a-00b1-4d77-bae5-d17b69e93685"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:52:43.408149 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.408130 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c323d6a-00b1-4d77-bae5-d17b69e93685-kube-api-access-9vw59" pod "2c323d6a-00b1-4d77-bae5-d17b69e93685" (UID: "2c323d6a-00b1-4d77-bae5-d17b69e93685"). InnerVolumeSpecName "kube-api-access-9vw59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:52:43.420834 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.420812 5019 generic.go:356] "Generic (PLEG): container finished" podID="db08f44d-3541-4f28-b6e5-bb5f16e937af" containerID="24d1c1920354ba2b9e2d9eed25227d09f2d37b7ad6cbe9d14117644a26126400" exitCode=0 Nov 03 18:52:43.420920 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.420853 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:43.420920 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.420911 5019 scope.go:122] "RemoveContainer" containerID="24d1c1920354ba2b9e2d9eed25227d09f2d37b7ad6cbe9d14117644a26126400" Nov 03 18:52:43.422278 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.422253 5019 generic.go:356] "Generic (PLEG): container finished" podID="58c3a579-8fbe-4a37-9917-40d0823a0899" containerID="e389d5c0da5230ca1c74e3ce764bbd66e324f0bd353e6852fefa37ad43644304" exitCode=0 Nov 03 18:52:43.422387 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.422300 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" Nov 03 18:52:43.423749 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.423730 5019 generic.go:356] "Generic (PLEG): container finished" podID="2c323d6a-00b1-4d77-bae5-d17b69e93685" containerID="1d31367b6e84f3e3feed8fd5f13d5221d2340ce34993bc6ad6a5625521468263" exitCode=0 Nov 03 18:52:43.423849 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.423766 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:43.430705 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.430688 5019 scope.go:122] "RemoveContainer" containerID="24d1c1920354ba2b9e2d9eed25227d09f2d37b7ad6cbe9d14117644a26126400" Nov 03 18:52:43.430973 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.430957 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d1c1920354ba2b9e2d9eed25227d09f2d37b7ad6cbe9d14117644a26126400\": container with ID starting with 24d1c1920354ba2b9e2d9eed25227d09f2d37b7ad6cbe9d14117644a26126400 not found: ID does not exist" containerID="24d1c1920354ba2b9e2d9eed25227d09f2d37b7ad6cbe9d14117644a26126400" Nov 03 18:52:43.431029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.430983 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d1c1920354ba2b9e2d9eed25227d09f2d37b7ad6cbe9d14117644a26126400"} err="failed to get container status \"24d1c1920354ba2b9e2d9eed25227d09f2d37b7ad6cbe9d14117644a26126400\": rpc error: code = NotFound desc = could not find container \"24d1c1920354ba2b9e2d9eed25227d09f2d37b7ad6cbe9d14117644a26126400\": container with ID starting with 24d1c1920354ba2b9e2d9eed25227d09f2d37b7ad6cbe9d14117644a26126400 not found: ID does not exist" Nov 03 18:52:43.431029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.430999 5019 scope.go:122] "RemoveContainer" containerID="e389d5c0da5230ca1c74e3ce764bbd66e324f0bd353e6852fefa37ad43644304" Nov 03 18:52:43.438391 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.438376 5019 scope.go:122] "RemoveContainer" containerID="e389d5c0da5230ca1c74e3ce764bbd66e324f0bd353e6852fefa37ad43644304" Nov 03 18:52:43.438667 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.438650 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e389d5c0da5230ca1c74e3ce764bbd66e324f0bd353e6852fefa37ad43644304\": container with ID starting with e389d5c0da5230ca1c74e3ce764bbd66e324f0bd353e6852fefa37ad43644304 not found: ID does not exist" containerID="e389d5c0da5230ca1c74e3ce764bbd66e324f0bd353e6852fefa37ad43644304" Nov 03 18:52:43.438718 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.438675 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e389d5c0da5230ca1c74e3ce764bbd66e324f0bd353e6852fefa37ad43644304"} err="failed to get container status \"e389d5c0da5230ca1c74e3ce764bbd66e324f0bd353e6852fefa37ad43644304\": rpc error: code = NotFound desc = could not find container \"e389d5c0da5230ca1c74e3ce764bbd66e324f0bd353e6852fefa37ad43644304\": container with ID starting with e389d5c0da5230ca1c74e3ce764bbd66e324f0bd353e6852fefa37ad43644304 not found: ID does not exist" Nov 03 18:52:43.438718 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.438693 5019 scope.go:122] "RemoveContainer" containerID="1d31367b6e84f3e3feed8fd5f13d5221d2340ce34993bc6ad6a5625521468263" Nov 03 18:52:43.445715 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.445697 5019 scope.go:122] "RemoveContainer" containerID="1d31367b6e84f3e3feed8fd5f13d5221d2340ce34993bc6ad6a5625521468263" Nov 03 18:52:43.445939 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.445923 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d31367b6e84f3e3feed8fd5f13d5221d2340ce34993bc6ad6a5625521468263\": container with ID starting with 1d31367b6e84f3e3feed8fd5f13d5221d2340ce34993bc6ad6a5625521468263 not found: ID does not exist" containerID="1d31367b6e84f3e3feed8fd5f13d5221d2340ce34993bc6ad6a5625521468263" Nov 03 18:52:43.445982 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.445946 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d31367b6e84f3e3feed8fd5f13d5221d2340ce34993bc6ad6a5625521468263"} err="failed to get container status \"1d31367b6e84f3e3feed8fd5f13d5221d2340ce34993bc6ad6a5625521468263\": rpc error: code = NotFound desc = could not find container \"1d31367b6e84f3e3feed8fd5f13d5221d2340ce34993bc6ad6a5625521468263\": container with ID starting with 1d31367b6e84f3e3feed8fd5f13d5221d2340ce34993bc6ad6a5625521468263 not found: ID does not exist" Nov 03 18:52:43.469738 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.469713 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-jlgwc\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="db08f44d-3541-4f28-b6e5-bb5f16e937af" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:43.490961 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.490912 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" Nov 03 18:52:43.499966 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.499947 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:52:43.507013 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.506866 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt" Nov 03 18:52:43.507013 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.506961 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db08f44d-3541-4f28-b6e5-bb5f16e937af-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db08f44d-3541-4f28-b6e5-bb5f16e937af-extensions-socket-volume\") pod \"db08f44d-3541-4f28-b6e5-bb5f16e937af\" (UID: \"db08f44d-3541-4f28-b6e5-bb5f16e937af\") " Nov 03 18:52:43.507159 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.507027 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c3a579-8fbe-4a37-9917-40d0823a0899-kube-api-access-z2zx6\" (UniqueName: \"kubernetes.io/projected/58c3a579-8fbe-4a37-9917-40d0823a0899-kube-api-access-z2zx6\") pod \"58c3a579-8fbe-4a37-9917-40d0823a0899\" (UID: \"58c3a579-8fbe-4a37-9917-40d0823a0899\") " Nov 03 18:52:43.507159 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.507077 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db08f44d-3541-4f28-b6e5-bb5f16e937af-kube-api-access-pznqr\" (UniqueName: \"kubernetes.io/projected/db08f44d-3541-4f28-b6e5-bb5f16e937af-kube-api-access-pznqr\") pod \"db08f44d-3541-4f28-b6e5-bb5f16e937af\" (UID: \"db08f44d-3541-4f28-b6e5-bb5f16e937af\") " Nov 03 18:52:43.507670 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.507362 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vw59\" (UniqueName: \"kubernetes.io/projected/2c323d6a-00b1-4d77-bae5-d17b69e93685-kube-api-access-9vw59\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:52:43.507670 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.507387 5019 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c323d6a-00b1-4d77-bae5-d17b69e93685-extensions-socket-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:52:43.508431 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.508159 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db08f44d-3541-4f28-b6e5-bb5f16e937af-extensions-socket-volume" pod "db08f44d-3541-4f28-b6e5-bb5f16e937af" (UID: "db08f44d-3541-4f28-b6e5-bb5f16e937af"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:52:43.510930 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.510906 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db08f44d-3541-4f28-b6e5-bb5f16e937af-kube-api-access-pznqr" pod "db08f44d-3541-4f28-b6e5-bb5f16e937af" (UID: "db08f44d-3541-4f28-b6e5-bb5f16e937af"). InnerVolumeSpecName "kube-api-access-pznqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:52:43.511929 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.511900 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c3a579-8fbe-4a37-9917-40d0823a0899-kube-api-access-z2zx6" pod "58c3a579-8fbe-4a37-9917-40d0823a0899" (UID: "58c3a579-8fbe-4a37-9917-40d0823a0899"). InnerVolumeSpecName "kube-api-access-z2zx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:52:43.607814 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.607790 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pznqr\" (UniqueName: \"kubernetes.io/projected/db08f44d-3541-4f28-b6e5-bb5f16e937af-kube-api-access-pznqr\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:52:43.607814 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.607815 5019 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db08f44d-3541-4f28-b6e5-bb5f16e937af-extensions-socket-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:52:43.607963 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.607826 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2zx6\" (UniqueName: \"kubernetes.io/projected/58c3a579-8fbe-4a37-9917-40d0823a0899-kube-api-access-z2zx6\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:52:43.628237 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.628193 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-vrc65\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="2c323d6a-00b1-4d77-bae5-d17b69e93685" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:43.675991 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.675955 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-vrc65\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="2c323d6a-00b1-4d77-bae5-d17b69e93685" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:43.723760 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.723722 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-hk4wd\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="58c3a579-8fbe-4a37-9917-40d0823a0899" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" Nov 03 18:52:43.771298 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.771241 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-jlgwc\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="db08f44d-3541-4f28-b6e5-bb5f16e937af" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:43.818990 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.818956 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-jlgwc\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="db08f44d-3541-4f28-b6e5-bb5f16e937af" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jlgwc" Nov 03 18:52:43.866577 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.866542 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-vrc65\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="2c323d6a-00b1-4d77-bae5-d17b69e93685" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vrc65" Nov 03 18:52:43.879822 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.879798 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj"] Nov 03 18:52:43.882230 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:52:43.882201 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a43aff_7fd6_4364_8db1_f819fd75da6c.slice/crio-35ad75794c54fb97d2c8c5c1b2b6c8b4eae451ede66885f7c7dddf61401e3c3d WatchSource:0}: Error finding container 35ad75794c54fb97d2c8c5c1b2b6c8b4eae451ede66885f7c7dddf61401e3c3d: Status 404 returned error can't find the container with id 35ad75794c54fb97d2c8c5c1b2b6c8b4eae451ede66885f7c7dddf61401e3c3d Nov 03 18:52:43.912250 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.912222 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt"] Nov 03 18:52:43.916828 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:43.916200 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws"] Nov 03 18:52:43.917714 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:52:43.917681 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-hk4wd\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="58c3a579-8fbe-4a37-9917-40d0823a0899" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hk4wd" Nov 03 18:52:43.931314 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:52:43.931292 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec280db_814f_478f_8824_22a20ec14aee.slice/crio-d9000dfa266c8d539a11bfc8e7affa0f072888fbe0ba95b7355b1bc68d493c55 WatchSource:0}: Error finding container d9000dfa266c8d539a11bfc8e7affa0f072888fbe0ba95b7355b1bc68d493c55: Status 404 returned error can't find the container with id d9000dfa266c8d539a11bfc8e7affa0f072888fbe0ba95b7355b1bc68d493c55 Nov 03 18:52:44.149036 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.148943 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c323d6a-00b1-4d77-bae5-d17b69e93685" path="/var/lib/kubelet/pods/2c323d6a-00b1-4d77-bae5-d17b69e93685/volumes" Nov 03 18:52:44.149541 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.149472 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c3a579-8fbe-4a37-9917-40d0823a0899" path="/var/lib/kubelet/pods/58c3a579-8fbe-4a37-9917-40d0823a0899/volumes" Nov 03 18:52:44.149859 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.149840 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db08f44d-3541-4f28-b6e5-bb5f16e937af" path="/var/lib/kubelet/pods/db08f44d-3541-4f28-b6e5-bb5f16e937af/volumes" Nov 03 18:52:44.432353 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.432309 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" event={"ID":"87a43aff-7fd6-4364-8db1-f819fd75da6c","Type":"ContainerStarted","Data":"484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372"} Nov 03 18:52:44.432536 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.432372 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" event={"ID":"87a43aff-7fd6-4364-8db1-f819fd75da6c","Type":"ContainerStarted","Data":"35ad75794c54fb97d2c8c5c1b2b6c8b4eae451ede66885f7c7dddf61401e3c3d"} Nov 03 18:52:44.432536 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.432427 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" Nov 03 18:52:44.434062 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.434038 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" event={"ID":"dec280db-814f-478f-8824-22a20ec14aee","Type":"ContainerStarted","Data":"f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768"} Nov 03 18:52:44.434183 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.434081 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" event={"ID":"dec280db-814f-478f-8824-22a20ec14aee","Type":"ContainerStarted","Data":"d9000dfa266c8d539a11bfc8e7affa0f072888fbe0ba95b7355b1bc68d493c55"} Nov 03 18:52:44.434183 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.434109 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:52:44.436387 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.436366 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt" event={"ID":"735187df-3794-4b77-a6f2-9ee7ea5803b4","Type":"ContainerStarted","Data":"f070395071d35eceecfcfc7b3bb731bc8fbbc51dc338d0f501a5e70e8508103e"} Nov 03 18:52:44.436475 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.436408 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt" event={"ID":"735187df-3794-4b77-a6f2-9ee7ea5803b4","Type":"ContainerStarted","Data":"c7e6f98f9b471dc878b54460313c6903806da8efb07e0e3b88c3d06265296df3"} Nov 03 18:52:44.436475 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.436461 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt" Nov 03 18:52:44.572546 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.572494 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" podStartSLOduration=2.572482014 podStartE2EDuration="2.572482014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:52:44.571440136 +0000 UTC m=+498.544001315" watchObservedRunningTime="2026-11-03 18:52:44.572482014 +0000 UTC m=+498.545043196" Nov 03 18:52:44.680375 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.680310 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt" podStartSLOduration=2.680300574 podStartE2EDuration="2.680300574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:52:44.680231658 +0000 UTC m=+498.652792840" watchObservedRunningTime="2026-11-03 18:52:44.680300574 +0000 UTC m=+498.652861756" Nov 03 18:52:44.793137 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:44.793038 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" podStartSLOduration=2.793024027 podStartE2EDuration="2.793024027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:52:44.791410056 +0000 UTC m=+498.763971241" watchObservedRunningTime="2026-11-03 18:52:44.793024027 +0000 UTC m=+498.765585209" Nov 03 18:52:54.486313 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:54.486238 5019 generic.go:356] "Generic (PLEG): container finished" podID="dec280db-814f-478f-8824-22a20ec14aee" containerID="f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768" exitCode=1 Nov 03 18:52:54.486788 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:54.486311 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" event={"ID":"dec280db-814f-478f-8824-22a20ec14aee","Type":"ContainerDied","Data":"f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768"} Nov 03 18:52:54.486788 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:54.486650 5019 scope.go:122] "RemoveContainer" containerID="f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768" Nov 03 18:52:55.442835 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:55.442804 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" Nov 03 18:52:55.443062 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:55.443032 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-2mzjt" Nov 03 18:52:55.492759 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:55.492726 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" event={"ID":"dec280db-814f-478f-8824-22a20ec14aee","Type":"ContainerStarted","Data":"a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d"} Nov 03 18:52:55.493095 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:52:55.492944 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:53:06.499902 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:06.499873 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:53:06.707793 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:06.707762 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj"] Nov 03 18:53:06.707973 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:06.707953 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" podUID="87a43aff-7fd6-4364-8db1-f819fd75da6c" containerName="manager" containerID="cri-o://484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372" gracePeriod=10 Nov 03 18:53:06.958110 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:06.958081 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" Nov 03 18:53:06.960596 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:06.960571 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr"] Nov 03 18:53:06.960927 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:06.960914 5019 cpu_manager.go:414] "containerMap: removing container" podUID="87a43aff-7fd6-4364-8db1-f819fd75da6c" containerName="manager" Nov 03 18:53:06.960927 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:06.960926 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a43aff-7fd6-4364-8db1-f819fd75da6c" containerName="manager" Nov 03 18:53:06.961052 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:06.961009 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a43aff-7fd6-4364-8db1-f819fd75da6c" containerName="manager" Nov 03 18:53:06.964095 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:06.964077 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" Nov 03 18:53:07.000430 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.000409 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a43aff-7fd6-4364-8db1-f819fd75da6c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87a43aff-7fd6-4364-8db1-f819fd75da6c-extensions-socket-volume\") pod \"87a43aff-7fd6-4364-8db1-f819fd75da6c\" (UID: \"87a43aff-7fd6-4364-8db1-f819fd75da6c\") " Nov 03 18:53:07.000528 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.000514 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a43aff-7fd6-4364-8db1-f819fd75da6c-kube-api-access-7t29r\" (UniqueName: \"kubernetes.io/projected/87a43aff-7fd6-4364-8db1-f819fd75da6c-kube-api-access-7t29r\") pod \"87a43aff-7fd6-4364-8db1-f819fd75da6c\" (UID: \"87a43aff-7fd6-4364-8db1-f819fd75da6c\") " Nov 03 18:53:07.000651 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.000631 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-xhdlr\" (UID: \"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" Nov 03 18:53:07.000723 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.000707 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsl6t\" (UniqueName: \"kubernetes.io/projected/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-kube-api-access-lsl6t\") pod \"kuadrant-operator-controller-manager-57bb9bfff-xhdlr\" (UID: \"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" Nov 03 18:53:07.000770 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.000744 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a43aff-7fd6-4364-8db1-f819fd75da6c-extensions-socket-volume" pod "87a43aff-7fd6-4364-8db1-f819fd75da6c" (UID: "87a43aff-7fd6-4364-8db1-f819fd75da6c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:53:07.000872 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.000851 5019 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87a43aff-7fd6-4364-8db1-f819fd75da6c-extensions-socket-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:53:07.002506 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.002477 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a43aff-7fd6-4364-8db1-f819fd75da6c-kube-api-access-7t29r" pod "87a43aff-7fd6-4364-8db1-f819fd75da6c" (UID: "87a43aff-7fd6-4364-8db1-f819fd75da6c"). InnerVolumeSpecName "kube-api-access-7t29r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:53:07.026530 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.026505 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr"] Nov 03 18:53:07.101556 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.101479 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-xhdlr\" (UID: \"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" Nov 03 18:53:07.101714 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.101636 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsl6t\" (UniqueName: \"kubernetes.io/projected/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-kube-api-access-lsl6t\") pod \"kuadrant-operator-controller-manager-57bb9bfff-xhdlr\" (UID: \"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" Nov 03 18:53:07.101714 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.101703 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t29r\" (UniqueName: \"kubernetes.io/projected/87a43aff-7fd6-4364-8db1-f819fd75da6c-kube-api-access-7t29r\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:53:07.101914 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.101891 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-xhdlr\" (UID: \"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" Nov 03 18:53:07.158849 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.158820 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsl6t\" (UniqueName: \"kubernetes.io/projected/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-kube-api-access-lsl6t\") pod \"kuadrant-operator-controller-manager-57bb9bfff-xhdlr\" (UID: \"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" Nov 03 18:53:07.276677 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.276643 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" Nov 03 18:53:07.556679 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.556639 5019 generic.go:356] "Generic (PLEG): container finished" podID="87a43aff-7fd6-4364-8db1-f819fd75da6c" containerID="484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372" exitCode=0 Nov 03 18:53:07.557071 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.556700 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" Nov 03 18:53:07.557071 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.556727 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" event={"ID":"87a43aff-7fd6-4364-8db1-f819fd75da6c","Type":"ContainerDied","Data":"484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372"} Nov 03 18:53:07.557071 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.556825 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj" event={"ID":"87a43aff-7fd6-4364-8db1-f819fd75da6c","Type":"ContainerDied","Data":"35ad75794c54fb97d2c8c5c1b2b6c8b4eae451ede66885f7c7dddf61401e3c3d"} Nov 03 18:53:07.557071 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.556854 5019 scope.go:122] "RemoveContainer" containerID="484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372" Nov 03 18:53:07.566159 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.566140 5019 scope.go:122] "RemoveContainer" containerID="484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372" Nov 03 18:53:07.566486 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:53:07.566467 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372\": container with ID starting with 484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372 not found: ID does not exist" containerID="484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372" Nov 03 18:53:07.566552 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.566494 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372"} err="failed to get container status \"484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372\": rpc error: code = NotFound desc = could not find container \"484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372\": container with ID starting with 484221d313b36be95dd6c50ec4089b40742a3e5fbbbb52e66ab4608c3f8d8372 not found: ID does not exist" Nov 03 18:53:07.688450 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.688369 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr"] Nov 03 18:53:07.694391 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:53:07.693082 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea9cb01_6cb5_45bf_bf93_2c00a3283b58.slice/crio-b929d88eaa9123a2e3d2168a73b63aa39e2444b1b9a8b48f50952342137312ba WatchSource:0}: Error finding container b929d88eaa9123a2e3d2168a73b63aa39e2444b1b9a8b48f50952342137312ba: Status 404 returned error can't find the container with id b929d88eaa9123a2e3d2168a73b63aa39e2444b1b9a8b48f50952342137312ba Nov 03 18:53:07.709972 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.709952 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj"] Nov 03 18:53:07.734188 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:07.734163 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kffzj"] Nov 03 18:53:08.148899 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:08.148857 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a43aff-7fd6-4364-8db1-f819fd75da6c" path="/var/lib/kubelet/pods/87a43aff-7fd6-4364-8db1-f819fd75da6c/volumes" Nov 03 18:53:08.563949 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:08.563913 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" event={"ID":"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58","Type":"ContainerStarted","Data":"c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514"} Nov 03 18:53:08.564431 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:08.563969 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" event={"ID":"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58","Type":"ContainerStarted","Data":"b929d88eaa9123a2e3d2168a73b63aa39e2444b1b9a8b48f50952342137312ba"} Nov 03 18:53:08.564431 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:08.564034 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" Nov 03 18:53:08.679401 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:08.679318 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" podStartSLOduration=2.679305039 podStartE2EDuration="2.679305039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:08.678774892 +0000 UTC m=+522.651336073" watchObservedRunningTime="2026-11-03 18:53:08.679305039 +0000 UTC m=+522.651866221" Nov 03 18:53:11.295097 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:53:11.295063 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:53:19.572062 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:19.572035 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" Nov 03 18:53:19.779918 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:19.779884 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws"] Nov 03 18:53:19.780108 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:19.780083 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" podUID="dec280db-814f-478f-8824-22a20ec14aee" containerName="manager" containerID="cri-o://a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d" gracePeriod=10 Nov 03 18:53:20.033624 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.033596 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:53:20.106070 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.106041 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec280db-814f-478f-8824-22a20ec14aee-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dec280db-814f-478f-8824-22a20ec14aee-extensions-socket-volume\") pod \"dec280db-814f-478f-8824-22a20ec14aee\" (UID: \"dec280db-814f-478f-8824-22a20ec14aee\") " Nov 03 18:53:20.106242 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.106090 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec280db-814f-478f-8824-22a20ec14aee-kube-api-access-hpnjx\" (UniqueName: \"kubernetes.io/projected/dec280db-814f-478f-8824-22a20ec14aee-kube-api-access-hpnjx\") pod \"dec280db-814f-478f-8824-22a20ec14aee\" (UID: \"dec280db-814f-478f-8824-22a20ec14aee\") " Nov 03 18:53:20.106449 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.106418 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec280db-814f-478f-8824-22a20ec14aee-extensions-socket-volume" pod "dec280db-814f-478f-8824-22a20ec14aee" (UID: "dec280db-814f-478f-8824-22a20ec14aee"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:53:20.108133 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.108110 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec280db-814f-478f-8824-22a20ec14aee-kube-api-access-hpnjx" pod "dec280db-814f-478f-8824-22a20ec14aee" (UID: "dec280db-814f-478f-8824-22a20ec14aee"). InnerVolumeSpecName "kube-api-access-hpnjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:53:20.207402 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.207354 5019 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dec280db-814f-478f-8824-22a20ec14aee-extensions-socket-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:53:20.207402 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.207388 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpnjx\" (UniqueName: \"kubernetes.io/projected/dec280db-814f-478f-8824-22a20ec14aee-kube-api-access-hpnjx\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:53:20.629467 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.629430 5019 generic.go:356] "Generic (PLEG): container finished" podID="dec280db-814f-478f-8824-22a20ec14aee" containerID="a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d" exitCode=0 Nov 03 18:53:20.629924 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.629489 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" Nov 03 18:53:20.629924 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.629487 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" event={"ID":"dec280db-814f-478f-8824-22a20ec14aee","Type":"ContainerDied","Data":"a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d"} Nov 03 18:53:20.629924 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.629635 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws" event={"ID":"dec280db-814f-478f-8824-22a20ec14aee","Type":"ContainerDied","Data":"d9000dfa266c8d539a11bfc8e7affa0f072888fbe0ba95b7355b1bc68d493c55"} Nov 03 18:53:20.629924 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.629667 5019 scope.go:122] "RemoveContainer" containerID="a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d" Nov 03 18:53:20.637880 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.637859 5019 scope.go:122] "RemoveContainer" containerID="f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768" Nov 03 18:53:20.645314 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.645301 5019 scope.go:122] "RemoveContainer" containerID="a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d" Nov 03 18:53:20.645615 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:53:20.645597 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d\": container with ID starting with a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d not found: ID does not exist" containerID="a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d" Nov 03 18:53:20.645663 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.645625 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d"} err="failed to get container status \"a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d\": rpc error: code = NotFound desc = could not find container \"a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d\": container with ID starting with a3e3810bf25147713c6e964df1766968b565ece787f7983316508a37658f3c4d not found: ID does not exist" Nov 03 18:53:20.645663 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.645648 5019 scope.go:122] "RemoveContainer" containerID="f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768" Nov 03 18:53:20.645876 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:53:20.645856 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768\": container with ID starting with f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768 not found: ID does not exist" containerID="f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768" Nov 03 18:53:20.645921 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.645880 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768"} err="failed to get container status \"f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768\": rpc error: code = NotFound desc = could not find container \"f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768\": container with ID starting with f42cd356d73a3ce28a75c5d5593d64ca04922610db27b0bf1ff888c9b5190768 not found: ID does not exist" Nov 03 18:53:20.832800 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.832769 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws"] Nov 03 18:53:20.879160 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:20.879129 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-m4sws"] Nov 03 18:53:22.148485 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:22.148452 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec280db-814f-478f-8824-22a20ec14aee" path="/var/lib/kubelet/pods/dec280db-814f-478f-8824-22a20ec14aee/volumes" Nov 03 18:53:29.799715 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.799681 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p5hpv"] Nov 03 18:53:29.800354 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.800092 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dec280db-814f-478f-8824-22a20ec14aee" containerName="manager" Nov 03 18:53:29.800354 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.800105 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec280db-814f-478f-8824-22a20ec14aee" containerName="manager" Nov 03 18:53:29.800354 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.800128 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dec280db-814f-478f-8824-22a20ec14aee" containerName="manager" Nov 03 18:53:29.800354 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.800133 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec280db-814f-478f-8824-22a20ec14aee" containerName="manager" Nov 03 18:53:29.800354 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.800192 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec280db-814f-478f-8824-22a20ec14aee" containerName="manager" Nov 03 18:53:29.804895 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.804874 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" Nov 03 18:53:29.853819 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.853793 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 18:53:29.861304 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.861283 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jfk6d\"" Nov 03 18:53:29.886684 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.886654 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tg2\" (UniqueName: \"kubernetes.io/projected/780a46d5-470b-4c98-90e2-b97af586f90b-kube-api-access-v2tg2\") pod \"limitador-limitador-74dfcfb894-p5hpv\" (UID: \"780a46d5-470b-4c98-90e2-b97af586f90b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" Nov 03 18:53:29.886805 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.886750 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/780a46d5-470b-4c98-90e2-b97af586f90b-config-file\") pod \"limitador-limitador-74dfcfb894-p5hpv\" (UID: \"780a46d5-470b-4c98-90e2-b97af586f90b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" Nov 03 18:53:29.901905 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.901881 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p5hpv"] Nov 03 18:53:29.987771 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.987744 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/780a46d5-470b-4c98-90e2-b97af586f90b-config-file\") pod \"limitador-limitador-74dfcfb894-p5hpv\" (UID: \"780a46d5-470b-4c98-90e2-b97af586f90b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" Nov 03 18:53:29.987897 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.987792 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tg2\" (UniqueName: \"kubernetes.io/projected/780a46d5-470b-4c98-90e2-b97af586f90b-kube-api-access-v2tg2\") pod \"limitador-limitador-74dfcfb894-p5hpv\" (UID: \"780a46d5-470b-4c98-90e2-b97af586f90b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" Nov 03 18:53:29.988415 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:29.988396 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/780a46d5-470b-4c98-90e2-b97af586f90b-config-file\") pod \"limitador-limitador-74dfcfb894-p5hpv\" (UID: \"780a46d5-470b-4c98-90e2-b97af586f90b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" Nov 03 18:53:30.045250 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:30.045219 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tg2\" (UniqueName: \"kubernetes.io/projected/780a46d5-470b-4c98-90e2-b97af586f90b-kube-api-access-v2tg2\") pod \"limitador-limitador-74dfcfb894-p5hpv\" (UID: \"780a46d5-470b-4c98-90e2-b97af586f90b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" Nov 03 18:53:30.051186 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:30.051128 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p5hpv"] Nov 03 18:53:30.116905 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:30.116877 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" Nov 03 18:53:30.482849 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:30.482825 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p5hpv"] Nov 03 18:53:30.485181 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:53:30.485032 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780a46d5_470b_4c98_90e2_b97af586f90b.slice/crio-999f99e0380a4f2b68f69be9feabf436f893f15386f25973f6611e0e1b15d33d WatchSource:0}: Error finding container 999f99e0380a4f2b68f69be9feabf436f893f15386f25973f6611e0e1b15d33d: Status 404 returned error can't find the container with id 999f99e0380a4f2b68f69be9feabf436f893f15386f25973f6611e0e1b15d33d Nov 03 18:53:30.675056 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:30.675019 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" event={"ID":"780a46d5-470b-4c98-90e2-b97af586f90b","Type":"ContainerStarted","Data":"999f99e0380a4f2b68f69be9feabf436f893f15386f25973f6611e0e1b15d33d"} Nov 03 18:53:32.685545 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:32.685517 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" event={"ID":"780a46d5-470b-4c98-90e2-b97af586f90b","Type":"ContainerStarted","Data":"00d14b3877e62ae031f7b7a88b52f2a49763bf720fbe772ab7f4eadd446b7579"} Nov 03 18:53:32.685858 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:32.685639 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" Nov 03 18:53:32.794654 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:32.794610 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" podStartSLOduration=1.681075151 podStartE2EDuration="3.79459558s" totalImagesPullingTime="2.113520429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:32.793300329 +0000 UTC m=+546.765861511" watchObservedRunningTime="2026-11-03 18:53:32.79459558 +0000 UTC m=+546.767156828" Nov 03 18:53:41.296076 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:53:41.296039 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:53:43.691277 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:43.691250 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" Nov 03 18:53:52.034271 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:52.034235 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p5hpv"] Nov 03 18:53:52.036735 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:52.034431 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" podUID="780a46d5-470b-4c98-90e2-b97af586f90b" containerName="limitador" containerID="cri-o://00d14b3877e62ae031f7b7a88b52f2a49763bf720fbe772ab7f4eadd446b7579" gracePeriod=30 Nov 03 18:53:52.775798 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:52.775763 5019 generic.go:356] "Generic (PLEG): container finished" podID="780a46d5-470b-4c98-90e2-b97af586f90b" containerID="00d14b3877e62ae031f7b7a88b52f2a49763bf720fbe772ab7f4eadd446b7579" exitCode=0 Nov 03 18:53:52.775961 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:52.775838 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" event={"ID":"780a46d5-470b-4c98-90e2-b97af586f90b","Type":"ContainerDied","Data":"00d14b3877e62ae031f7b7a88b52f2a49763bf720fbe772ab7f4eadd446b7579"} Nov 03 18:53:52.975546 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:52.975523 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" Nov 03 18:53:53.091023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:53.090946 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780a46d5-470b-4c98-90e2-b97af586f90b-config-file\" (UniqueName: \"kubernetes.io/configmap/780a46d5-470b-4c98-90e2-b97af586f90b-config-file\") pod \"780a46d5-470b-4c98-90e2-b97af586f90b\" (UID: \"780a46d5-470b-4c98-90e2-b97af586f90b\") " Nov 03 18:53:53.091023 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:53.090990 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780a46d5-470b-4c98-90e2-b97af586f90b-kube-api-access-v2tg2\" (UniqueName: \"kubernetes.io/projected/780a46d5-470b-4c98-90e2-b97af586f90b-kube-api-access-v2tg2\") pod \"780a46d5-470b-4c98-90e2-b97af586f90b\" (UID: \"780a46d5-470b-4c98-90e2-b97af586f90b\") " Nov 03 18:53:53.091561 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:53.091381 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780a46d5-470b-4c98-90e2-b97af586f90b-config-file" pod "780a46d5-470b-4c98-90e2-b97af586f90b" (UID: "780a46d5-470b-4c98-90e2-b97af586f90b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:53:53.093182 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:53.093158 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780a46d5-470b-4c98-90e2-b97af586f90b-kube-api-access-v2tg2" pod "780a46d5-470b-4c98-90e2-b97af586f90b" (UID: "780a46d5-470b-4c98-90e2-b97af586f90b"). InnerVolumeSpecName "kube-api-access-v2tg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:53:53.192551 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:53.192508 5019 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/780a46d5-470b-4c98-90e2-b97af586f90b-config-file\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:53:53.192551 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:53.192547 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2tg2\" (UniqueName: \"kubernetes.io/projected/780a46d5-470b-4c98-90e2-b97af586f90b-kube-api-access-v2tg2\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:53:53.781782 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:53.781750 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" event={"ID":"780a46d5-470b-4c98-90e2-b97af586f90b","Type":"ContainerDied","Data":"999f99e0380a4f2b68f69be9feabf436f893f15386f25973f6611e0e1b15d33d"} Nov 03 18:53:53.781782 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:53.781774 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-p5hpv" Nov 03 18:53:53.782027 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:53.781818 5019 scope.go:122] "RemoveContainer" containerID="00d14b3877e62ae031f7b7a88b52f2a49763bf720fbe772ab7f4eadd446b7579" Nov 03 18:53:53.935745 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:53.935713 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p5hpv"] Nov 03 18:53:53.982231 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:53.982202 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p5hpv"] Nov 03 18:53:54.149076 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:53:54.149001 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780a46d5-470b-4c98-90e2-b97af586f90b" path="/var/lib/kubelet/pods/780a46d5-470b-4c98-90e2-b97af586f90b/volumes" Nov 03 18:54:00.111852 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.111818 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j"] Nov 03 18:54:00.112248 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.112181 5019 cpu_manager.go:414] "containerMap: removing container" podUID="780a46d5-470b-4c98-90e2-b97af586f90b" containerName="limitador" Nov 03 18:54:00.112248 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.112191 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780a46d5-470b-4c98-90e2-b97af586f90b" containerName="limitador" Nov 03 18:54:00.112317 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.112263 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="780a46d5-470b-4c98-90e2-b97af586f90b" containerName="limitador" Nov 03 18:54:00.112317 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.112273 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec280db-814f-478f-8824-22a20ec14aee" containerName="manager" Nov 03 18:54:00.116848 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.116827 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" Nov 03 18:54:00.212162 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.212125 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 18:54:00.212162 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.212143 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:54:00.212162 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.212125 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:54:00.212410 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.212192 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-f5t22\"" Nov 03 18:54:00.213703 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.213684 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j"] Nov 03 18:54:00.255644 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.255609 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrs7n\" (UniqueName: \"kubernetes.io/projected/550a4d2f-33de-42bc-9a90-d89dbc7771b7-kube-api-access-xrs7n\") pod \"postgres-7cdbf6dc44-r6l7j\" (UID: \"550a4d2f-33de-42bc-9a90-d89dbc7771b7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" Nov 03 18:54:00.255808 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.255682 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/550a4d2f-33de-42bc-9a90-d89dbc7771b7-data\") pod \"postgres-7cdbf6dc44-r6l7j\" (UID: \"550a4d2f-33de-42bc-9a90-d89dbc7771b7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" Nov 03 18:54:00.356274 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.356242 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/550a4d2f-33de-42bc-9a90-d89dbc7771b7-data\") pod \"postgres-7cdbf6dc44-r6l7j\" (UID: \"550a4d2f-33de-42bc-9a90-d89dbc7771b7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" Nov 03 18:54:00.356460 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.356350 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrs7n\" (UniqueName: \"kubernetes.io/projected/550a4d2f-33de-42bc-9a90-d89dbc7771b7-kube-api-access-xrs7n\") pod \"postgres-7cdbf6dc44-r6l7j\" (UID: \"550a4d2f-33de-42bc-9a90-d89dbc7771b7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" Nov 03 18:54:00.356643 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.356625 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/550a4d2f-33de-42bc-9a90-d89dbc7771b7-data\") pod \"postgres-7cdbf6dc44-r6l7j\" (UID: \"550a4d2f-33de-42bc-9a90-d89dbc7771b7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" Nov 03 18:54:00.413969 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.413903 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrs7n\" (UniqueName: \"kubernetes.io/projected/550a4d2f-33de-42bc-9a90-d89dbc7771b7-kube-api-access-xrs7n\") pod \"postgres-7cdbf6dc44-r6l7j\" (UID: \"550a4d2f-33de-42bc-9a90-d89dbc7771b7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" Nov 03 18:54:00.429731 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.429706 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" Nov 03 18:54:00.793774 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.793753 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j"] Nov 03 18:54:00.796087 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:54:00.796053 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550a4d2f_33de_42bc_9a90_d89dbc7771b7.slice/crio-6e0389fb8ec346f73e744c4275fa855f934c98d75c64cc9aeb8ec1012c84e435 WatchSource:0}: Error finding container 6e0389fb8ec346f73e744c4275fa855f934c98d75c64cc9aeb8ec1012c84e435: Status 404 returned error can't find the container with id 6e0389fb8ec346f73e744c4275fa855f934c98d75c64cc9aeb8ec1012c84e435 Nov 03 18:54:00.815048 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:00.815017 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" event={"ID":"550a4d2f-33de-42bc-9a90-d89dbc7771b7","Type":"ContainerStarted","Data":"6e0389fb8ec346f73e744c4275fa855f934c98d75c64cc9aeb8ec1012c84e435"} Nov 03 18:54:09.865407 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:09.865354 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" event={"ID":"550a4d2f-33de-42bc-9a90-d89dbc7771b7","Type":"ContainerStarted","Data":"ca2ebb7e18c6a58c2dbf1ea97cdce9cd6417eaf60e4db11b647ba560b19d3a8d"} Nov 03 18:54:09.865922 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:09.865425 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" Nov 03 18:54:09.975121 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:09.975051 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" podStartSLOduration=1.839920386 podStartE2EDuration="9.975031649s" totalImagesPullingTime="8.135111263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:09.973145702 +0000 UTC m=+583.945706884" watchObservedRunningTime="2026-11-03 18:54:09.975031649 +0000 UTC m=+583.947592832" Nov 03 18:54:11.296597 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:54:11.296556 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:54:15.899535 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:15.899507 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r6l7j" Nov 03 18:54:26.193926 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:26.193899 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 18:54:26.194401 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:26.193949 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 18:54:27.083978 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.083946 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-ldvsg"] Nov 03 18:54:27.108624 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.108598 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:27.157544 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.157523 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 18:54:27.157544 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.157528 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 18:54:27.164685 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.164664 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hdw97\"" Nov 03 18:54:27.171695 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.171562 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-webhook-cert\") pod \"maas-controller-577f596567-ldvsg\" (UID: \"3137f313-ce17-433e-a741-e84d7f475c75\") " pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:27.171695 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.171689 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-metrics-certs\") pod \"maas-controller-577f596567-ldvsg\" (UID: \"3137f313-ce17-433e-a741-e84d7f475c75\") " pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:27.171884 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.171751 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhlck\" (UniqueName: \"kubernetes.io/projected/3137f313-ce17-433e-a741-e84d7f475c75-kube-api-access-dhlck\") pod \"maas-controller-577f596567-ldvsg\" (UID: \"3137f313-ce17-433e-a741-e84d7f475c75\") " pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:27.203951 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.203925 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 18:54:27.205664 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.205642 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-ldvsg"] Nov 03 18:54:27.272660 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.272631 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhlck\" (UniqueName: \"kubernetes.io/projected/3137f313-ce17-433e-a741-e84d7f475c75-kube-api-access-dhlck\") pod \"maas-controller-577f596567-ldvsg\" (UID: \"3137f313-ce17-433e-a741-e84d7f475c75\") " pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:27.272828 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.272671 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-webhook-cert\") pod \"maas-controller-577f596567-ldvsg\" (UID: \"3137f313-ce17-433e-a741-e84d7f475c75\") " pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:27.272894 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.272858 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-metrics-certs\") pod \"maas-controller-577f596567-ldvsg\" (UID: \"3137f313-ce17-433e-a741-e84d7f475c75\") " pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:27.275299 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.275272 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-metrics-certs\") pod \"maas-controller-577f596567-ldvsg\" (UID: \"3137f313-ce17-433e-a741-e84d7f475c75\") " pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:27.275424 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.275272 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-webhook-cert\") pod \"maas-controller-577f596567-ldvsg\" (UID: \"3137f313-ce17-433e-a741-e84d7f475c75\") " pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:27.329882 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.329855 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhlck\" (UniqueName: \"kubernetes.io/projected/3137f313-ce17-433e-a741-e84d7f475c75-kube-api-access-dhlck\") pod \"maas-controller-577f596567-ldvsg\" (UID: \"3137f313-ce17-433e-a741-e84d7f475c75\") " pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:27.421521 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.421498 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:27.610432 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.610396 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bbcd964c6-q7xqh"] Nov 03 18:54:27.615259 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.615238 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:27.672375 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.672306 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbcd964c6-q7xqh"] Nov 03 18:54:27.675968 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.675940 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-webhook-cert\") pod \"maas-controller-bbcd964c6-q7xqh\" (UID: \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\") " pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:27.676080 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.676024 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-metrics-certs\") pod \"maas-controller-bbcd964c6-q7xqh\" (UID: \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\") " pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:27.676196 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.676170 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjxd9\" (UniqueName: \"kubernetes.io/projected/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-kube-api-access-xjxd9\") pod \"maas-controller-bbcd964c6-q7xqh\" (UID: \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\") " pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:27.776924 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.776891 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjxd9\" (UniqueName: \"kubernetes.io/projected/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-kube-api-access-xjxd9\") pod \"maas-controller-bbcd964c6-q7xqh\" (UID: \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\") " pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:27.777059 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.776940 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-webhook-cert\") pod \"maas-controller-bbcd964c6-q7xqh\" (UID: \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\") " pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:27.777059 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.776987 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-metrics-certs\") pod \"maas-controller-bbcd964c6-q7xqh\" (UID: \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\") " pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:27.779429 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.779376 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-metrics-certs\") pod \"maas-controller-bbcd964c6-q7xqh\" (UID: \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\") " pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:27.779429 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.779410 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-webhook-cert\") pod \"maas-controller-bbcd964c6-q7xqh\" (UID: \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\") " pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:27.830721 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.830699 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-ldvsg"] Nov 03 18:54:27.832860 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:54:27.832826 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3137f313_ce17_433e_a741_e84d7f475c75.slice/crio-f31f2bb2ff1d41967dec72d3a84319f7970af93484e4ace622edf88bcb440981 WatchSource:0}: Error finding container f31f2bb2ff1d41967dec72d3a84319f7970af93484e4ace622edf88bcb440981: Status 404 returned error can't find the container with id f31f2bb2ff1d41967dec72d3a84319f7970af93484e4ace622edf88bcb440981 Nov 03 18:54:27.836730 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.836708 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjxd9\" (UniqueName: \"kubernetes.io/projected/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-kube-api-access-xjxd9\") pod \"maas-controller-bbcd964c6-q7xqh\" (UID: \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\") " pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:27.929163 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.929076 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:27.945135 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:27.945108 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-ldvsg" event={"ID":"3137f313-ce17-433e-a741-e84d7f475c75","Type":"ContainerStarted","Data":"f31f2bb2ff1d41967dec72d3a84319f7970af93484e4ace622edf88bcb440981"} Nov 03 18:54:28.294473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:28.294393 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbcd964c6-q7xqh"] Nov 03 18:54:28.298381 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:54:28.298350 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbea2cc2_3245_4a60_8af0_1bf0292f7b62.slice/crio-c8eeb05f85b613e4ac04970af2521a698a85a02c38d4892ff633086653500e99 WatchSource:0}: Error finding container c8eeb05f85b613e4ac04970af2521a698a85a02c38d4892ff633086653500e99: Status 404 returned error can't find the container with id c8eeb05f85b613e4ac04970af2521a698a85a02c38d4892ff633086653500e99 Nov 03 18:54:28.950401 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:28.950362 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbcd964c6-q7xqh" event={"ID":"cbea2cc2-3245-4a60-8af0-1bf0292f7b62","Type":"ContainerStarted","Data":"c8eeb05f85b613e4ac04970af2521a698a85a02c38d4892ff633086653500e99"} Nov 03 18:54:33.981532 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:33.981492 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-ldvsg" event={"ID":"3137f313-ce17-433e-a741-e84d7f475c75","Type":"ContainerStarted","Data":"7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239"} Nov 03 18:54:33.982005 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:33.981623 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:33.983279 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:33.983253 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbcd964c6-q7xqh" event={"ID":"cbea2cc2-3245-4a60-8af0-1bf0292f7b62","Type":"ContainerStarted","Data":"1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6"} Nov 03 18:54:33.983425 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:33.983359 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:34.093161 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:34.093112 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-ldvsg" podStartSLOduration=1.5301829919999999 podStartE2EDuration="7.093099558s" totalImagesPullingTime="5.562916566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:34.091521511 +0000 UTC m=+608.064082724" watchObservedRunningTime="2026-11-03 18:54:34.093099558 +0000 UTC m=+608.065660740" Nov 03 18:54:34.202420 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:34.202370 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bbcd964c6-q7xqh" podStartSLOduration=2.09631981 podStartE2EDuration="7.2023547s" totalImagesPullingTime="5.10603489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:34.201115569 +0000 UTC m=+608.173676763" watchObservedRunningTime="2026-11-03 18:54:34.2023547 +0000 UTC m=+608.174915908" Nov 03 18:54:38.598151 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.598112 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv"] Nov 03 18:54:38.601984 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.601964 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:38.679342 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.679307 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:38.679563 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.679456 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:38.679642 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.679585 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:38.679694 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.679657 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrjmd\" (UniqueName: \"kubernetes.io/projected/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-kube-api-access-zrjmd\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:38.697214 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.697187 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 18:54:38.697214 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.697200 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 18:54:38.697463 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.697269 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 18:54:38.697463 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.697320 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-zwjn6\"" Nov 03 18:54:38.699146 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.699107 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv"] Nov 03 18:54:38.780969 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.780939 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:38.781123 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.780995 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:38.781123 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.781037 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:38.781123 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.781078 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjmd\" (UniqueName: \"kubernetes.io/projected/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-kube-api-access-zrjmd\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:38.781250 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:54:38.781127 5019 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 18:54:38.781250 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:54:38.781157 5019 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 03 18:54:38.781250 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:54:38.781201 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-proxy-source podName:e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f nodeName:}" failed. No retries permitted until 2026-11-03 18:54:39.281183059 +0000 UTC m=+613.253744238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-proxy-source") pod "usage-logs-tenancy-proxy-7d94f55f4b-msvrv" (UID: "e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f") : configmap "usage-tenancy-proxy-source" not found Nov 03 18:54:38.781250 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:54:38.781215 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-tls-cert podName:e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f nodeName:}" failed. No retries permitted until 2026-11-03 18:54:39.281209575 +0000 UTC m=+613.253770734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-tls-cert") pod "usage-logs-tenancy-proxy-7d94f55f4b-msvrv" (UID: "e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f") : secret "usage-logs-tenancy-proxy-tls" not found Nov 03 18:54:38.781584 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.781567 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:38.838531 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:38.838499 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjmd\" (UniqueName: \"kubernetes.io/projected/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-kube-api-access-zrjmd\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:39.286573 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:39.286542 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:39.286761 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:39.286662 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:39.287338 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:39.287310 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:39.289109 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:39.289090 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-msvrv\" (UID: \"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:39.515127 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:39.515094 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:39.880292 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:39.880269 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv"] Nov 03 18:54:39.882357 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:54:39.882299 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2be4b9a_ec7c_4c2c_a3d2_4cce86c9a74f.slice/crio-f084934636769d438081c03e3ccd22fb6747154827fb64c052dfc5a5c10628c4 WatchSource:0}: Error finding container f084934636769d438081c03e3ccd22fb6747154827fb64c052dfc5a5c10628c4: Status 404 returned error can't find the container with id f084934636769d438081c03e3ccd22fb6747154827fb64c052dfc5a5c10628c4 Nov 03 18:54:40.022099 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:40.022057 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" event={"ID":"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f","Type":"ContainerStarted","Data":"f084934636769d438081c03e3ccd22fb6747154827fb64c052dfc5a5c10628c4"} Nov 03 18:54:41.296242 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:54:41.296205 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:54:44.990382 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:44.990344 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 18:54:44.991264 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:44.991245 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:45.150998 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:45.150964 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-ldvsg"] Nov 03 18:54:45.151204 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:45.151156 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-ldvsg" podUID="3137f313-ce17-433e-a741-e84d7f475c75" containerName="manager" containerID="cri-o://7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239" gracePeriod=30 Nov 03 18:54:45.457529 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:45.457502 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:45.542114 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:45.540980 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-metrics-certs\") pod \"3137f313-ce17-433e-a741-e84d7f475c75\" (UID: \"3137f313-ce17-433e-a741-e84d7f475c75\") " Nov 03 18:54:45.542114 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:45.541488 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-webhook-cert\") pod \"3137f313-ce17-433e-a741-e84d7f475c75\" (UID: \"3137f313-ce17-433e-a741-e84d7f475c75\") " Nov 03 18:54:45.542114 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:45.541607 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3137f313-ce17-433e-a741-e84d7f475c75-kube-api-access-dhlck\" (UniqueName: \"kubernetes.io/projected/3137f313-ce17-433e-a741-e84d7f475c75-kube-api-access-dhlck\") pod \"3137f313-ce17-433e-a741-e84d7f475c75\" (UID: \"3137f313-ce17-433e-a741-e84d7f475c75\") " Nov 03 18:54:45.544647 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:45.544612 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-webhook-cert" pod "3137f313-ce17-433e-a741-e84d7f475c75" (UID: "3137f313-ce17-433e-a741-e84d7f475c75"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:54:45.544777 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:45.544720 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3137f313-ce17-433e-a741-e84d7f475c75-kube-api-access-dhlck" pod "3137f313-ce17-433e-a741-e84d7f475c75" (UID: "3137f313-ce17-433e-a741-e84d7f475c75"). InnerVolumeSpecName "kube-api-access-dhlck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:54:45.546160 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:45.546131 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-metrics-certs" pod "3137f313-ce17-433e-a741-e84d7f475c75" (UID: "3137f313-ce17-433e-a741-e84d7f475c75"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:54:45.642814 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:45.642776 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:54:45.642814 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:45.642811 5019 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3137f313-ce17-433e-a741-e84d7f475c75-webhook-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:54:45.643053 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:45.642828 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhlck\" (UniqueName: \"kubernetes.io/projected/3137f313-ce17-433e-a741-e84d7f475c75-kube-api-access-dhlck\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:54:46.054629 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:46.054481 5019 generic.go:356] "Generic (PLEG): container finished" podID="3137f313-ce17-433e-a741-e84d7f475c75" containerID="7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239" exitCode=0 Nov 03 18:54:46.054629 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:46.054568 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-ldvsg" Nov 03 18:54:46.054629 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:46.054572 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-ldvsg" event={"ID":"3137f313-ce17-433e-a741-e84d7f475c75","Type":"ContainerDied","Data":"7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239"} Nov 03 18:54:46.055218 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:46.054672 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-ldvsg" event={"ID":"3137f313-ce17-433e-a741-e84d7f475c75","Type":"ContainerDied","Data":"f31f2bb2ff1d41967dec72d3a84319f7970af93484e4ace622edf88bcb440981"} Nov 03 18:54:46.055218 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:46.054704 5019 scope.go:122] "RemoveContainer" containerID="7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239" Nov 03 18:54:46.070685 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:46.070663 5019 scope.go:122] "RemoveContainer" containerID="7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239" Nov 03 18:54:46.070944 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:54:46.070919 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239\": container with ID starting with 7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239 not found: ID does not exist" containerID="7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239" Nov 03 18:54:46.071024 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:46.070956 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239"} err="failed to get container status \"7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239\": rpc error: code = NotFound desc = could not find container \"7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239\": container with ID starting with 7964c9adab5f74dec083b365ea3c3b33f0305e2757de1e97cc68a64988767239 not found: ID does not exist" Nov 03 18:54:46.210832 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:46.210795 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-ldvsg"] Nov 03 18:54:46.256970 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:46.256937 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-ldvsg"] Nov 03 18:54:48.150973 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:48.150938 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3137f313-ce17-433e-a741-e84d7f475c75" path="/var/lib/kubelet/pods/3137f313-ce17-433e-a741-e84d7f475c75/volumes" Nov 03 18:54:54.098693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:54.098659 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" event={"ID":"e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f","Type":"ContainerStarted","Data":"666b590c3bc1323153ab74d22984ab2b2e03764c10e43d4b4f9642c1088b3f3f"} Nov 03 18:54:54.099064 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:54.098727 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:54:54.208525 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:54:54.208471 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" podStartSLOduration=2.084923619 podStartE2EDuration="16.20845807s" totalImagesPullingTime="14.123534451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:54.207781505 +0000 UTC m=+628.180342708" watchObservedRunningTime="2026-11-03 18:54:54.20845807 +0000 UTC m=+628.181019251" Nov 03 18:55:05.109069 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.109032 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-msvrv" Nov 03 18:55:05.840913 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.840877 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hpds8"] Nov 03 18:55:05.841382 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.841335 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3137f313-ce17-433e-a741-e84d7f475c75" containerName="manager" Nov 03 18:55:05.841382 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.841352 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3137f313-ce17-433e-a741-e84d7f475c75" containerName="manager" Nov 03 18:55:05.841481 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.841437 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3137f313-ce17-433e-a741-e84d7f475c75" containerName="manager" Nov 03 18:55:05.850222 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.850193 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:05.923877 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.923842 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-maas-api-tls\") pod \"maas-api-5c48877c75-hpds8\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:05.923877 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.923877 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-metrics-certs\") pod \"maas-api-5c48877c75-hpds8\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:05.924109 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.923906 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjfgb\" (UniqueName: \"kubernetes.io/projected/552330ce-c90c-42a5-b783-effafa6afb8f-kube-api-access-gjfgb\") pod \"maas-api-5c48877c75-hpds8\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:05.924109 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.924048 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552330ce-c90c-42a5-b783-effafa6afb8f-openshift-service-ca\") pod \"maas-api-5c48877c75-hpds8\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:05.945794 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.945766 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zdsbl\"" Nov 03 18:55:05.945794 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.945768 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 18:55:05.945971 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.945831 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 18:55:05.947421 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:05.947401 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hpds8"] Nov 03 18:55:06.024613 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:06.024583 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-maas-api-tls\") pod \"maas-api-5c48877c75-hpds8\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:06.024613 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:06.024612 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-metrics-certs\") pod \"maas-api-5c48877c75-hpds8\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:06.024820 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:06.024640 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfgb\" (UniqueName: \"kubernetes.io/projected/552330ce-c90c-42a5-b783-effafa6afb8f-kube-api-access-gjfgb\") pod \"maas-api-5c48877c75-hpds8\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:06.024820 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:06.024745 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552330ce-c90c-42a5-b783-effafa6afb8f-openshift-service-ca\") pod \"maas-api-5c48877c75-hpds8\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:06.025383 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:06.025358 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552330ce-c90c-42a5-b783-effafa6afb8f-openshift-service-ca\") pod \"maas-api-5c48877c75-hpds8\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:06.027083 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:06.027058 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-maas-api-tls\") pod \"maas-api-5c48877c75-hpds8\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:06.027180 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:06.027090 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-metrics-certs\") pod \"maas-api-5c48877c75-hpds8\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:06.081675 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:06.081647 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjfgb\" (UniqueName: \"kubernetes.io/projected/552330ce-c90c-42a5-b783-effafa6afb8f-kube-api-access-gjfgb\") pod \"maas-api-5c48877c75-hpds8\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:06.163309 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:06.163239 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:06.530246 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:06.530220 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hpds8"] Nov 03 18:55:06.532896 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:55:06.532867 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552330ce_c90c_42a5_b783_effafa6afb8f.slice/crio-4a88ebfc3d5c3a164b979fe65e095547fae121b0072d4b5ea90f9611ee48ae74 WatchSource:0}: Error finding container 4a88ebfc3d5c3a164b979fe65e095547fae121b0072d4b5ea90f9611ee48ae74: Status 404 returned error can't find the container with id 4a88ebfc3d5c3a164b979fe65e095547fae121b0072d4b5ea90f9611ee48ae74 Nov 03 18:55:07.158022 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:07.157990 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" event={"ID":"552330ce-c90c-42a5-b783-effafa6afb8f","Type":"ContainerStarted","Data":"4a88ebfc3d5c3a164b979fe65e095547fae121b0072d4b5ea90f9611ee48ae74"} Nov 03 18:55:11.301746 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:55:11.301702 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:55:12.183667 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:12.183634 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" event={"ID":"552330ce-c90c-42a5-b783-effafa6afb8f","Type":"ContainerStarted","Data":"1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48"} Nov 03 18:55:12.183820 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:12.183769 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:12.294143 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:12.294095 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" podStartSLOduration=2.274224057 podStartE2EDuration="7.294082718s" totalImagesPullingTime="5.019858661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:55:12.293479187 +0000 UTC m=+646.266040369" watchObservedRunningTime="2026-11-03 18:55:12.294082718 +0000 UTC m=+646.266643900" Nov 03 18:55:18.193864 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:55:18.193836 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 18:55:41.295892 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:55:41.295813 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:56:11.295203 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:56:11.295168 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:56:22.442112 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.442074 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn"] Nov 03 18:56:22.445673 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.445653 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.488623 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.488589 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64142b5e-3faa-403f-a7b7-60638cfeb7fc-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.488770 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.488648 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64142b5e-3faa-403f-a7b7-60638cfeb7fc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.488868 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.488842 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64142b5e-3faa-403f-a7b7-60638cfeb7fc-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.488922 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.488903 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64142b5e-3faa-403f-a7b7-60638cfeb7fc-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.489007 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.488987 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64142b5e-3faa-403f-a7b7-60638cfeb7fc-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.489068 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.489039 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzxd\" (UniqueName: \"kubernetes.io/projected/64142b5e-3faa-403f-a7b7-60638cfeb7fc-kube-api-access-qkzxd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.540934 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.540902 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 18:56:22.541144 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.541000 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 18:56:22.541144 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.541059 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-z8pj6\"" Nov 03 18:56:22.541239 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.541140 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 18:56:22.542632 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.542613 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn"] Nov 03 18:56:22.596035 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.596001 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64142b5e-3faa-403f-a7b7-60638cfeb7fc-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.596279 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.596261 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64142b5e-3faa-403f-a7b7-60638cfeb7fc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.596541 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.596519 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64142b5e-3faa-403f-a7b7-60638cfeb7fc-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.596670 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.596577 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64142b5e-3faa-403f-a7b7-60638cfeb7fc-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.596949 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.596926 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64142b5e-3faa-403f-a7b7-60638cfeb7fc-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.597069 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.597043 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzxd\" (UniqueName: \"kubernetes.io/projected/64142b5e-3faa-403f-a7b7-60638cfeb7fc-kube-api-access-qkzxd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.597136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.597049 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64142b5e-3faa-403f-a7b7-60638cfeb7fc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.597136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.597110 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64142b5e-3faa-403f-a7b7-60638cfeb7fc-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.597216 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.597177 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64142b5e-3faa-403f-a7b7-60638cfeb7fc-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.599707 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.599676 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64142b5e-3faa-403f-a7b7-60638cfeb7fc-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.607496 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.607470 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64142b5e-3faa-403f-a7b7-60638cfeb7fc-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.653967 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.653940 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzxd\" (UniqueName: \"kubernetes.io/projected/64142b5e-3faa-403f-a7b7-60638cfeb7fc-kube-api-access-qkzxd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lzknn\" (UID: \"64142b5e-3faa-403f-a7b7-60638cfeb7fc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:22.758184 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:22.758103 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:23.121855 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:23.121674 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn"] Nov 03 18:56:23.124416 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:56:23.124388 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64142b5e_3faa_403f_a7b7_60638cfeb7fc.slice/crio-2b0ce4dd3e6a44e79c52f2e277f97888bf6586bad47259b426fa5b73c3939061 WatchSource:0}: Error finding container 2b0ce4dd3e6a44e79c52f2e277f97888bf6586bad47259b426fa5b73c3939061: Status 404 returned error can't find the container with id 2b0ce4dd3e6a44e79c52f2e277f97888bf6586bad47259b426fa5b73c3939061 Nov 03 18:56:23.513535 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:23.513505 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" event={"ID":"64142b5e-3faa-403f-a7b7-60638cfeb7fc","Type":"ContainerStarted","Data":"2b0ce4dd3e6a44e79c52f2e277f97888bf6586bad47259b426fa5b73c3939061"} Nov 03 18:56:25.525902 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:25.525860 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" event={"ID":"64142b5e-3faa-403f-a7b7-60638cfeb7fc","Type":"ContainerStarted","Data":"b3a5d13520fe59f6bd526938055385575c3a8c23d58ec7ac4d5902209f58bc05"} Nov 03 18:56:25.636239 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:25.636188 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" podStartSLOduration=2.062136908 podStartE2EDuration="3.636174969s" totalImagesPullingTime="1.574038061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:25.635564157 +0000 UTC m=+719.608125338" watchObservedRunningTime="2026-11-03 18:56:25.636174969 +0000 UTC m=+719.608736151" Nov 03 18:56:25.758866 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:25.758838 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:26.887219 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:26.887190 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc"] Nov 03 18:56:26.890923 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:26.890894 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:26.986152 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:26.986117 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 18:56:26.987991 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:26.987965 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc"] Nov 03 18:56:27.036960 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.036909 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.037136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.037043 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.037136 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.037112 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.037257 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.037152 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d27w\" (UniqueName: \"kubernetes.io/projected/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-kube-api-access-2d27w\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.037257 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.037194 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.037398 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.037299 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.137746 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.137678 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.137746 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.137708 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.137958 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.137748 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.137958 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.137773 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.137958 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.137795 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d27w\" (UniqueName: \"kubernetes.io/projected/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-kube-api-access-2d27w\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.137958 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.137835 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.138157 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.138086 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.138221 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.138160 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.138281 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.138248 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.140007 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.139985 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.140416 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.140396 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.195252 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.195226 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d27w\" (UniqueName: \"kubernetes.io/projected/fc9e70c5-7751-4377-aacf-3c8c8530bdc9-kube-api-access-2d27w\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vspbc\" (UID: \"fc9e70c5-7751-4377-aacf-3c8c8530bdc9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.204068 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.204048 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:27.566524 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:27.566497 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc"] Nov 03 18:56:27.568541 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:56:27.568512 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9e70c5_7751_4377_aacf_3c8c8530bdc9.slice/crio-5e75a58297d77b746d93537e51196823e223fbe028a97234a1742e44785aeb61 WatchSource:0}: Error finding container 5e75a58297d77b746d93537e51196823e223fbe028a97234a1742e44785aeb61: Status 404 returned error can't find the container with id 5e75a58297d77b746d93537e51196823e223fbe028a97234a1742e44785aeb61 Nov 03 18:56:28.541213 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:28.541128 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" event={"ID":"fc9e70c5-7751-4377-aacf-3c8c8530bdc9","Type":"ContainerStarted","Data":"782303b57de7f7d0b1b7b93b98b9f7829d35d64e7ce10b62a4e20c420889a556"} Nov 03 18:56:28.541213 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:28.541185 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" event={"ID":"fc9e70c5-7751-4377-aacf-3c8c8530bdc9","Type":"ContainerStarted","Data":"5e75a58297d77b746d93537e51196823e223fbe028a97234a1742e44785aeb61"} Nov 03 18:56:28.652847 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:28.652797 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" podStartSLOduration=2.032458212 podStartE2EDuration="2.652782619s" totalImagesPullingTime="620.324407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:28.651465196 +0000 UTC m=+722.624026378" watchObservedRunningTime="2026-11-03 18:56:28.652782619 +0000 UTC m=+722.625343853" Nov 03 18:56:29.204424 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:29.204383 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:32.758935 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:32.758900 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:32.772090 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:32.772066 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:33.574720 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:33.574693 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" Nov 03 18:56:34.292997 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.292969 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp"] Nov 03 18:56:34.296773 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.296753 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.345679 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.345648 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 18:56:34.393571 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.393543 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp"] Nov 03 18:56:34.403541 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.403522 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.403616 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.403559 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.403616 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.403588 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.403720 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.403639 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcv94\" (UniqueName: \"kubernetes.io/projected/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-kube-api-access-fcv94\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.403720 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.403675 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.403720 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.403706 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.505207 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.505174 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.505425 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.505214 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.505425 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.505235 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.505425 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.505254 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcv94\" (UniqueName: \"kubernetes.io/projected/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-kube-api-access-fcv94\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.505425 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.505276 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.505425 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.505314 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.505743 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.505719 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.505795 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.505762 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.505795 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.505783 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.507545 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.507526 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.507725 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.507709 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.562364 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.562281 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcv94\" (UniqueName: \"kubernetes.io/projected/2704fa8b-f7ed-49ec-918c-45d4df1cc1dc-kube-api-access-fcv94\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp\" (UID: \"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.609802 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.609779 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:34.973434 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:34.973405 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp"] Nov 03 18:56:34.975482 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:56:34.975417 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2704fa8b_f7ed_49ec_918c_45d4df1cc1dc.slice/crio-42fdfd2b4dd86c7c1d8f3e7a4fd79b2a4aa846deba502b017440770cdd2e72b4 WatchSource:0}: Error finding container 42fdfd2b4dd86c7c1d8f3e7a4fd79b2a4aa846deba502b017440770cdd2e72b4: Status 404 returned error can't find the container with id 42fdfd2b4dd86c7c1d8f3e7a4fd79b2a4aa846deba502b017440770cdd2e72b4 Nov 03 18:56:35.575621 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:35.575585 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" event={"ID":"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc","Type":"ContainerStarted","Data":"42fdfd2b4dd86c7c1d8f3e7a4fd79b2a4aa846deba502b017440770cdd2e72b4"} Nov 03 18:56:36.582592 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:36.582555 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" event={"ID":"2704fa8b-f7ed-49ec-918c-45d4df1cc1dc","Type":"ContainerStarted","Data":"67a9abc784d4a4b229a7f37798b6fb92244207be7d18a743a1a555a25803855b"} Nov 03 18:56:36.610069 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:36.610031 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:36.694283 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:36.694231 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" podStartSLOduration=1.978711356 podStartE2EDuration="2.694217246s" totalImagesPullingTime="715.50589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:36.693140189 +0000 UTC m=+730.665701370" watchObservedRunningTime="2026-11-03 18:56:36.694217246 +0000 UTC m=+730.666778427" Nov 03 18:56:37.204720 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:37.204681 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:37.217981 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:37.217950 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:37.598299 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:37.598214 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" Nov 03 18:56:41.295942 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:56:41.295909 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:56:44.610284 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:44.610239 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:44.623830 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:44.623803 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:44.635470 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:44.635447 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp" Nov 03 18:56:56.374298 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.374260 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs"] Nov 03 18:56:56.380662 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.380640 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.394531 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.394507 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.394642 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.394612 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.394704 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.394652 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.394789 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.394773 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5740ed16-bc9e-46d7-8157-5267888355d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.394849 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.394811 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87nn\" (UniqueName: \"kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-kube-api-access-c87nn\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.394916 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.394892 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.395045 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.395021 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5740ed16-bc9e-46d7-8157-5267888355d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.395097 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.395082 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.395187 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.395109 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.420999 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.420978 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g"] Nov 03 18:56:56.424797 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.424783 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.435249 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.435229 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-26cvs\"" Nov 03 18:56:56.480973 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.480934 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs"] Nov 03 18:56:56.490964 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.490934 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g"] Nov 03 18:56:56.495958 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.495929 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87nn\" (UniqueName: \"kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-kube-api-access-c87nn\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.496103 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.495974 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.496103 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.495998 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3355b2e-2808-446c-be03-26729210bfb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.496103 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.496028 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.496103 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.496057 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5740ed16-bc9e-46d7-8157-5267888355d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.496103 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.496085 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95jnn\" (UniqueName: \"kubernetes.io/projected/b3355b2e-2808-446c-be03-26729210bfb6-kube-api-access-95jnn\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.496403 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.496244 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.496403 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.496378 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.497729 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.496281 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.497835 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.497761 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.497835 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.497792 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3355b2e-2808-446c-be03-26729210bfb6-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.497951 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.497839 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.497951 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.497874 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.498059 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.497950 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.498059 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.497981 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3355b2e-2808-446c-be03-26729210bfb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.498059 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.498016 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.498059 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.498048 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.498272 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.498149 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.498272 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.498177 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5740ed16-bc9e-46d7-8157-5267888355d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.498272 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.498259 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.498630 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.498606 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.498845 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.498825 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5740ed16-bc9e-46d7-8157-5267888355d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.498941 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.498925 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.499124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.499105 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5740ed16-bc9e-46d7-8157-5267888355d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.500353 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.500315 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.554973 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.554941 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.555118 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.555060 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87nn\" (UniqueName: \"kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-kube-api-access-c87nn\") pod \"maas-default-gateway-openshift-default-77f556d647-z89fs\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.599626 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.599586 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95jnn\" (UniqueName: \"kubernetes.io/projected/b3355b2e-2808-446c-be03-26729210bfb6-kube-api-access-95jnn\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.599802 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.599638 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.599802 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.599671 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3355b2e-2808-446c-be03-26729210bfb6-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.599802 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.599714 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.599802 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.599766 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.599802 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.599794 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3355b2e-2808-446c-be03-26729210bfb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.600170 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.599871 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.600170 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.599931 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3355b2e-2808-446c-be03-26729210bfb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.600170 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.599964 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.600356 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.600232 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.600974 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.600843 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3355b2e-2808-446c-be03-26729210bfb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.600974 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.600881 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.600974 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.600929 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.601474 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.601429 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.603553 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.603530 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3355b2e-2808-446c-be03-26729210bfb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.605418 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.605399 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3355b2e-2808-446c-be03-26729210bfb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.657382 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.657293 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95jnn\" (UniqueName: \"kubernetes.io/projected/b3355b2e-2808-446c-be03-26729210bfb6-kube-api-access-95jnn\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.657772 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.657745 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3355b2e-2808-446c-be03-26729210bfb6-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-c699g\" (UID: \"b3355b2e-2808-446c-be03-26729210bfb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:56.694841 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.694812 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:56:56.739522 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:56.739490 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:56:57.111612 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:57.111584 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs"] Nov 03 18:56:57.113681 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:56:57.113651 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5740ed16_bc9e_46d7_8157_5267888355d6.slice/crio-9c68c0b81e627ebd98bf7eca002e07f6935ec0a8138bfd4ccae364c015d0ce55 WatchSource:0}: Error finding container 9c68c0b81e627ebd98bf7eca002e07f6935ec0a8138bfd4ccae364c015d0ce55: Status 404 returned error can't find the container with id 9c68c0b81e627ebd98bf7eca002e07f6935ec0a8138bfd4ccae364c015d0ce55 Nov 03 18:56:57.114870 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:57.114850 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g"] Nov 03 18:56:57.116945 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:56:57.116923 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3355b2e_2808_446c_be03_26729210bfb6.slice/crio-685f5bd70de13ded5320ac98c9e5712b502919b1d0f623cd3dd9bf357318e5b8 WatchSource:0}: Error finding container 685f5bd70de13ded5320ac98c9e5712b502919b1d0f623cd3dd9bf357318e5b8: Status 404 returned error can't find the container with id 685f5bd70de13ded5320ac98c9e5712b502919b1d0f623cd3dd9bf357318e5b8 Nov 03 18:56:57.685080 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:57.685043 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" event={"ID":"b3355b2e-2808-446c-be03-26729210bfb6","Type":"ContainerStarted","Data":"685f5bd70de13ded5320ac98c9e5712b502919b1d0f623cd3dd9bf357318e5b8"} Nov 03 18:56:57.686261 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:56:57.686232 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" event={"ID":"5740ed16-bc9e-46d7-8157-5267888355d6","Type":"ContainerStarted","Data":"9c68c0b81e627ebd98bf7eca002e07f6935ec0a8138bfd4ccae364c015d0ce55"} Nov 03 18:57:01.127224 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:01.127190 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:57:01.127527 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:01.127259 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:57:01.127527 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:01.127289 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:57:01.135914 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:01.135883 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:57:01.136026 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:01.135946 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:57:01.136026 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:01.135979 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:57:01.717607 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:01.717567 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" event={"ID":"b3355b2e-2808-446c-be03-26729210bfb6","Type":"ContainerStarted","Data":"5efc340a12fe0cc29d319fd04fd357e58a144290859e289b6a25192030408395"} Nov 03 18:57:01.719382 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:01.719353 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" event={"ID":"5740ed16-bc9e-46d7-8157-5267888355d6","Type":"ContainerStarted","Data":"a71ecfba7a1fd457dcc08d5f82aec0919103d2433074224eee61bec2dcfdd3df"} Nov 03 18:57:01.739760 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:01.739737 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:57:01.930225 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:01.930159 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" podStartSLOduration=1.91406386 podStartE2EDuration="5.930144486s" totalImagesPullingTime="4.016080626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:01.883593943 +0000 UTC m=+755.856155126" watchObservedRunningTime="2026-11-03 18:57:01.930144486 +0000 UTC m=+755.902705668" Nov 03 18:57:02.005408 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:02.004892 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" podStartSLOduration=1.9942807 podStartE2EDuration="6.004871966s" totalImagesPullingTime="4.010591266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:02.002632732 +0000 UTC m=+755.975193917" watchObservedRunningTime="2026-11-03 18:57:02.004871966 +0000 UTC m=+755.977433147" Nov 03 18:57:02.697379 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:02.697346 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:57:02.698044 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:02.698018 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z89fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:57:02.698284 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:02.698258 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" podUID="5740ed16-bc9e-46d7-8157-5267888355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 18:57:02.741186 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:02.741156 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-c699g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 18:57:02.741434 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:02.741413 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" podUID="b3355b2e-2808-446c-be03-26729210bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 18:57:03.701141 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:03.700683 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z89fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:57:03.701141 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:03.700746 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" podUID="5740ed16-bc9e-46d7-8157-5267888355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 18:57:03.740437 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:03.740394 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-c699g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 18:57:03.740626 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:03.740454 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" podUID="b3355b2e-2808-446c-be03-26729210bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 18:57:03.920971 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:03.920935 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7489558957-b4tmh"] Nov 03 18:57:03.925095 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:03.925075 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" Nov 03 18:57:04.002586 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.002503 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7489558957-b4tmh"] Nov 03 18:57:04.018901 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.018858 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9557w\" (UniqueName: \"kubernetes.io/projected/0e9f360a-938f-45c6-9960-844bafd0ecff-kube-api-access-9557w\") pod \"ai-gateway-controller-7489558957-b4tmh\" (UID: \"0e9f360a-938f-45c6-9960-844bafd0ecff\") " pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" Nov 03 18:57:04.019130 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.019110 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e9f360a-938f-45c6-9960-844bafd0ecff-metrics-certs\") pod \"ai-gateway-controller-7489558957-b4tmh\" (UID: \"0e9f360a-938f-45c6-9960-844bafd0ecff\") " pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" Nov 03 18:57:04.032442 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.032408 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 18:57:04.032631 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.032471 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-xh7t2\"" Nov 03 18:57:04.032708 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.032568 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 18:57:04.120214 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.120177 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9557w\" (UniqueName: \"kubernetes.io/projected/0e9f360a-938f-45c6-9960-844bafd0ecff-kube-api-access-9557w\") pod \"ai-gateway-controller-7489558957-b4tmh\" (UID: \"0e9f360a-938f-45c6-9960-844bafd0ecff\") " pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" Nov 03 18:57:04.120214 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.120220 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e9f360a-938f-45c6-9960-844bafd0ecff-metrics-certs\") pod \"ai-gateway-controller-7489558957-b4tmh\" (UID: \"0e9f360a-938f-45c6-9960-844bafd0ecff\") " pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" Nov 03 18:57:04.126041 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.126012 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e9f360a-938f-45c6-9960-844bafd0ecff-metrics-certs\") pod \"ai-gateway-controller-7489558957-b4tmh\" (UID: \"0e9f360a-938f-45c6-9960-844bafd0ecff\") " pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" Nov 03 18:57:04.204254 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.204172 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9557w\" (UniqueName: \"kubernetes.io/projected/0e9f360a-938f-45c6-9960-844bafd0ecff-kube-api-access-9557w\") pod \"ai-gateway-controller-7489558957-b4tmh\" (UID: \"0e9f360a-938f-45c6-9960-844bafd0ecff\") " pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" Nov 03 18:57:04.240030 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.239799 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" Nov 03 18:57:04.653113 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.653039 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7489558957-b4tmh"] Nov 03 18:57:04.657512 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:57:04.657482 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9f360a_938f_45c6_9960_844bafd0ecff.slice/crio-be5e702e16014845d73223e0a2f7232857d54b0800094b7dce601434a3a53c77 WatchSource:0}: Error finding container be5e702e16014845d73223e0a2f7232857d54b0800094b7dce601434a3a53c77: Status 404 returned error can't find the container with id be5e702e16014845d73223e0a2f7232857d54b0800094b7dce601434a3a53c77 Nov 03 18:57:04.658843 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.658821 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:57:04.696738 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.696704 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z89fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:57:04.697006 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.696974 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" podUID="5740ed16-bc9e-46d7-8157-5267888355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 18:57:04.735546 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.735510 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" event={"ID":"0e9f360a-938f-45c6-9960-844bafd0ecff","Type":"ContainerStarted","Data":"be5e702e16014845d73223e0a2f7232857d54b0800094b7dce601434a3a53c77"} Nov 03 18:57:04.740043 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.740013 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-c699g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 18:57:04.740160 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:04.740062 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" podUID="b3355b2e-2808-446c-be03-26729210bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 18:57:05.702416 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:05.698870 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z89fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:57:05.702416 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:05.698947 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" podUID="5740ed16-bc9e-46d7-8157-5267888355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 18:57:05.739914 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:05.739880 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-c699g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 18:57:05.740672 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:05.740638 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" podUID="b3355b2e-2808-446c-be03-26729210bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 18:57:06.699351 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:06.698699 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z89fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:57:06.699351 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:06.698760 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" podUID="5740ed16-bc9e-46d7-8157-5267888355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 18:57:06.740899 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:06.740803 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-c699g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 18:57:06.741297 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:06.740874 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" podUID="b3355b2e-2808-446c-be03-26729210bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 18:57:07.695505 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:07.695467 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z89fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:57:07.695682 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:07.695539 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" podUID="5740ed16-bc9e-46d7-8157-5267888355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 18:57:07.740029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:07.739988 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-c699g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 18:57:07.740199 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:07.740054 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" podUID="b3355b2e-2808-446c-be03-26729210bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 18:57:08.695967 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:08.695930 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z89fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:57:08.696524 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:08.695985 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" podUID="5740ed16-bc9e-46d7-8157-5267888355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 18:57:08.740605 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:08.740571 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-c699g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 18:57:08.740766 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:08.740617 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" podUID="b3355b2e-2808-446c-be03-26729210bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 18:57:09.897686 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:09.897649 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:57:09.898168 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:09.898128 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:57:09.899488 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:09.899462 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 18:57:09.992256 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:09.992212 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:57:09.993450 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:09.993380 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:57:09.994932 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:09.994909 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-c699g" Nov 03 18:57:10.774359 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:10.774306 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" event={"ID":"0e9f360a-938f-45c6-9960-844bafd0ecff","Type":"ContainerStarted","Data":"081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7"} Nov 03 18:57:10.886970 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:10.886877 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" podStartSLOduration=1.9783055649999999 podStartE2EDuration="7.886864427s" totalImagesPullingTime="5.908558862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:10.884801712 +0000 UTC m=+764.857362897" watchObservedRunningTime="2026-11-03 18:57:10.886864427 +0000 UTC m=+764.859425664" Nov 03 18:57:11.296450 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:57:11.296417 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:57:11.779614 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:11.779578 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" Nov 03 18:57:15.155506 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.155475 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-txjmz"] Nov 03 18:57:15.159250 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.159232 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" Nov 03 18:57:15.239557 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.239519 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db160175-01ea-4325-b1e7-60249341c899-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-txjmz\" (UID: \"db160175-01ea-4325-b1e7-60249341c899\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" Nov 03 18:57:15.239557 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.239580 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txhch\" (UniqueName: \"kubernetes.io/projected/db160175-01ea-4325-b1e7-60249341c899-kube-api-access-txhch\") pod \"payload-pre-processing-7b98bb678c-txjmz\" (UID: \"db160175-01ea-4325-b1e7-60249341c899\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" Nov 03 18:57:15.264812 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.254486 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8skxv\"" Nov 03 18:57:15.264812 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.254486 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 18:57:15.264812 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.258459 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-txjmz"] Nov 03 18:57:15.340530 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.340490 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db160175-01ea-4325-b1e7-60249341c899-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-txjmz\" (UID: \"db160175-01ea-4325-b1e7-60249341c899\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" Nov 03 18:57:15.340738 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.340568 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txhch\" (UniqueName: \"kubernetes.io/projected/db160175-01ea-4325-b1e7-60249341c899-kube-api-access-txhch\") pod \"payload-pre-processing-7b98bb678c-txjmz\" (UID: \"db160175-01ea-4325-b1e7-60249341c899\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" Nov 03 18:57:15.341235 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.341213 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db160175-01ea-4325-b1e7-60249341c899-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-txjmz\" (UID: \"db160175-01ea-4325-b1e7-60249341c899\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" Nov 03 18:57:15.398560 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.398520 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txhch\" (UniqueName: \"kubernetes.io/projected/db160175-01ea-4325-b1e7-60249341c899-kube-api-access-txhch\") pod \"payload-pre-processing-7b98bb678c-txjmz\" (UID: \"db160175-01ea-4325-b1e7-60249341c899\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" Nov 03 18:57:15.471854 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.471827 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" Nov 03 18:57:15.843541 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:15.843506 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-txjmz"] Nov 03 18:57:15.845262 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:57:15.845229 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb160175_01ea_4325_b1e7_60249341c899.slice/crio-79a8978b28a2b9e6fd13c7ab26b62567b21aaab52d269ee82347c6b02fa1fe9e WatchSource:0}: Error finding container 79a8978b28a2b9e6fd13c7ab26b62567b21aaab52d269ee82347c6b02fa1fe9e: Status 404 returned error can't find the container with id 79a8978b28a2b9e6fd13c7ab26b62567b21aaab52d269ee82347c6b02fa1fe9e Nov 03 18:57:16.808830 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:16.808651 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" event={"ID":"db160175-01ea-4325-b1e7-60249341c899","Type":"ContainerStarted","Data":"79a8978b28a2b9e6fd13c7ab26b62567b21aaab52d269ee82347c6b02fa1fe9e"} Nov 03 18:57:19.827201 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:19.827161 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" event={"ID":"db160175-01ea-4325-b1e7-60249341c899","Type":"ContainerStarted","Data":"ac1fbdaf4abd4ae5e135a42559fe72cb352477a63f037880adf65a7931b7a057"} Nov 03 18:57:19.827681 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:19.827223 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" Nov 03 18:57:19.936147 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:19.936092 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" podStartSLOduration=1.8846000950000001 podStartE2EDuration="4.936079248s" totalImagesPullingTime="3.051479153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:19.934403921 +0000 UTC m=+773.906965104" watchObservedRunningTime="2026-11-03 18:57:19.936079248 +0000 UTC m=+773.908640430" Nov 03 18:57:22.786298 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:22.786269 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" Nov 03 18:57:23.124077 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:23.124001 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r"] Nov 03 18:57:23.128056 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:23.128030 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" Nov 03 18:57:23.185010 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:23.184980 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r"] Nov 03 18:57:23.215153 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:23.215121 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grmd2\" (UniqueName: \"kubernetes.io/projected/8cdf44f6-ad95-4a5c-b11f-73c70208f9ec-kube-api-access-grmd2\") pod \"ai-gateway-controller-7d4c5c4549-hsf4r\" (UID: \"8cdf44f6-ad95-4a5c-b11f-73c70208f9ec\") " pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" Nov 03 18:57:23.215396 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:23.215264 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cdf44f6-ad95-4a5c-b11f-73c70208f9ec-metrics-certs\") pod \"ai-gateway-controller-7d4c5c4549-hsf4r\" (UID: \"8cdf44f6-ad95-4a5c-b11f-73c70208f9ec\") " pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" Nov 03 18:57:23.316969 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:23.316940 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grmd2\" (UniqueName: \"kubernetes.io/projected/8cdf44f6-ad95-4a5c-b11f-73c70208f9ec-kube-api-access-grmd2\") pod \"ai-gateway-controller-7d4c5c4549-hsf4r\" (UID: \"8cdf44f6-ad95-4a5c-b11f-73c70208f9ec\") " pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" Nov 03 18:57:23.317124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:23.317003 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cdf44f6-ad95-4a5c-b11f-73c70208f9ec-metrics-certs\") pod \"ai-gateway-controller-7d4c5c4549-hsf4r\" (UID: \"8cdf44f6-ad95-4a5c-b11f-73c70208f9ec\") " pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" Nov 03 18:57:23.319359 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:23.319315 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cdf44f6-ad95-4a5c-b11f-73c70208f9ec-metrics-certs\") pod \"ai-gateway-controller-7d4c5c4549-hsf4r\" (UID: \"8cdf44f6-ad95-4a5c-b11f-73c70208f9ec\") " pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" Nov 03 18:57:23.374150 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:23.374089 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grmd2\" (UniqueName: \"kubernetes.io/projected/8cdf44f6-ad95-4a5c-b11f-73c70208f9ec-kube-api-access-grmd2\") pod \"ai-gateway-controller-7d4c5c4549-hsf4r\" (UID: \"8cdf44f6-ad95-4a5c-b11f-73c70208f9ec\") " pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" Nov 03 18:57:23.442065 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:23.442040 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" Nov 03 18:57:23.805472 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:23.805450 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r"] Nov 03 18:57:23.807047 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:57:23.807019 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdf44f6_ad95_4a5c_b11f_73c70208f9ec.slice/crio-b130bf0e9cf505d29a4894f1aa1a5fe443e505ed22c3dd402fdde1b9b4969626 WatchSource:0}: Error finding container b130bf0e9cf505d29a4894f1aa1a5fe443e505ed22c3dd402fdde1b9b4969626: Status 404 returned error can't find the container with id b130bf0e9cf505d29a4894f1aa1a5fe443e505ed22c3dd402fdde1b9b4969626 Nov 03 18:57:23.848828 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:23.848802 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" event={"ID":"8cdf44f6-ad95-4a5c-b11f-73c70208f9ec","Type":"ContainerStarted","Data":"b130bf0e9cf505d29a4894f1aa1a5fe443e505ed22c3dd402fdde1b9b4969626"} Nov 03 18:57:24.854756 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:24.854722 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" event={"ID":"8cdf44f6-ad95-4a5c-b11f-73c70208f9ec","Type":"ContainerStarted","Data":"c763f689de10cd999eab5551d6721e4dd0171b8e63a283ed83962f757b1d2a14"} Nov 03 18:57:24.855099 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:24.854834 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" Nov 03 18:57:24.965890 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:24.965840 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" podStartSLOduration=1.011766157 podStartE2EDuration="1.965827276s" totalImagesPullingTime="954.061119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:24.964526875 +0000 UTC m=+778.937088058" watchObservedRunningTime="2026-11-03 18:57:24.965827276 +0000 UTC m=+778.938388504" Nov 03 18:57:30.836002 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:30.835971 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txjmz" Nov 03 18:57:35.862585 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:35.862556 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d4c5c4549-hsf4r" Nov 03 18:57:36.019452 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.019414 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7489558957-b4tmh"] Nov 03 18:57:36.019692 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.019664 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" podUID="0e9f360a-938f-45c6-9960-844bafd0ecff" containerName="manager" containerID="cri-o://081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7" gracePeriod=30 Nov 03 18:57:36.279231 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.279206 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" Nov 03 18:57:36.434286 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.434253 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e9f360a-938f-45c6-9960-844bafd0ecff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e9f360a-938f-45c6-9960-844bafd0ecff-metrics-certs\") pod \"0e9f360a-938f-45c6-9960-844bafd0ecff\" (UID: \"0e9f360a-938f-45c6-9960-844bafd0ecff\") " Nov 03 18:57:36.434468 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.434410 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9f360a-938f-45c6-9960-844bafd0ecff-kube-api-access-9557w\" (UniqueName: \"kubernetes.io/projected/0e9f360a-938f-45c6-9960-844bafd0ecff-kube-api-access-9557w\") pod \"0e9f360a-938f-45c6-9960-844bafd0ecff\" (UID: \"0e9f360a-938f-45c6-9960-844bafd0ecff\") " Nov 03 18:57:36.436292 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.436266 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9f360a-938f-45c6-9960-844bafd0ecff-metrics-certs" pod "0e9f360a-938f-45c6-9960-844bafd0ecff" (UID: "0e9f360a-938f-45c6-9960-844bafd0ecff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:57:36.436380 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.436322 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9f360a-938f-45c6-9960-844bafd0ecff-kube-api-access-9557w" pod "0e9f360a-938f-45c6-9960-844bafd0ecff" (UID: "0e9f360a-938f-45c6-9960-844bafd0ecff"). InnerVolumeSpecName "kube-api-access-9557w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:57:36.536046 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.536015 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9557w\" (UniqueName: \"kubernetes.io/projected/0e9f360a-938f-45c6-9960-844bafd0ecff-kube-api-access-9557w\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:36.536046 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.536041 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e9f360a-938f-45c6-9960-844bafd0ecff-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:36.914559 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.914471 5019 generic.go:356] "Generic (PLEG): container finished" podID="0e9f360a-938f-45c6-9960-844bafd0ecff" containerID="081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7" exitCode=0 Nov 03 18:57:36.914559 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.914518 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" Nov 03 18:57:36.915075 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.914522 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" event={"ID":"0e9f360a-938f-45c6-9960-844bafd0ecff","Type":"ContainerDied","Data":"081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7"} Nov 03 18:57:36.915075 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.914639 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7489558957-b4tmh" event={"ID":"0e9f360a-938f-45c6-9960-844bafd0ecff","Type":"ContainerDied","Data":"be5e702e16014845d73223e0a2f7232857d54b0800094b7dce601434a3a53c77"} Nov 03 18:57:36.915075 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.914662 5019 scope.go:122] "RemoveContainer" containerID="081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7" Nov 03 18:57:36.924055 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.924035 5019 scope.go:122] "RemoveContainer" containerID="081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7" Nov 03 18:57:36.924312 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:57:36.924291 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7\": container with ID starting with 081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7 not found: ID does not exist" containerID="081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7" Nov 03 18:57:36.924425 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:36.924317 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7"} err="failed to get container status \"081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7\": rpc error: code = NotFound desc = could not find container \"081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7\": container with ID starting with 081a9b9713ce210a7a8e823a1a32f3da976cdcb0e893a70150ce4b88076b8aa7 not found: ID does not exist" Nov 03 18:57:37.067797 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:37.067770 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7489558957-b4tmh"] Nov 03 18:57:37.073418 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:37.073393 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7489558957-b4tmh"] Nov 03 18:57:38.149201 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:57:38.149159 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9f360a-938f-45c6-9960-844bafd0ecff" path="/var/lib/kubelet/pods/0e9f360a-938f-45c6-9960-844bafd0ecff/volumes" Nov 03 18:57:41.296869 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:57:41.296833 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:58:08.616488 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.616457 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh"] Nov 03 18:58:08.617089 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.616942 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0e9f360a-938f-45c6-9960-844bafd0ecff" containerName="manager" Nov 03 18:58:08.617089 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.616956 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9f360a-938f-45c6-9960-844bafd0ecff" containerName="manager" Nov 03 18:58:08.617089 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.617055 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9f360a-938f-45c6-9960-844bafd0ecff" containerName="manager" Nov 03 18:58:08.624432 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.624408 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.673205 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.673168 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fa2878-openshift-default-dockercfg-hfvcf\"" Nov 03 18:58:08.723289 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.723252 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh"] Nov 03 18:58:08.734655 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.734627 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.734837 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.734665 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9874\" (UniqueName: \"kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-kube-api-access-t9874\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.734837 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.734701 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107848ae-227c-4b41-ab04-19b284e22fd3-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.734958 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.734855 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107848ae-227c-4b41-ab04-19b284e22fd3-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.734958 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.734899 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.735068 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.734975 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.735068 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.735028 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.735068 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.735051 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.735221 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.735076 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.835891 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.835860 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107848ae-227c-4b41-ab04-19b284e22fd3-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836074 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.835903 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107848ae-227c-4b41-ab04-19b284e22fd3-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836074 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.835929 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836168 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.836065 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836168 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.836151 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836294 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.836192 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836294 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.836227 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836495 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.836300 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836495 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.836362 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9874\" (UniqueName: \"kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-kube-api-access-t9874\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836495 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.836380 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836716 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.836591 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836716 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.836704 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107848ae-227c-4b41-ab04-19b284e22fd3-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836811 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.836748 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.836846 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.836827 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.838681 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.838618 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.838681 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.838611 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107848ae-227c-4b41-ab04-19b284e22fd3-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.900687 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.900620 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.900828 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.900749 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9874\" (UniqueName: \"kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-kube-api-access-t9874\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:08.938451 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:08.938422 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:09.257276 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.257245 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc"] Nov 03 18:58:09.261153 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.261128 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.310102 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.310075 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-90a9bf-openshift-default-dockercfg-jqm4m\"" Nov 03 18:58:09.322694 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.322666 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc"] Nov 03 18:58:09.342393 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.342350 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.342538 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.342410 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa2f0da-f709-4e28-a544-e7ff20f78633-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.342538 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.342489 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-589zn\" (UniqueName: \"kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-kube-api-access-589zn\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.342619 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.342545 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.342692 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.342667 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.342777 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.342762 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.342838 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.342812 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.342902 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.342885 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.342941 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.342919 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.364824 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.364801 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh"] Nov 03 18:58:09.368739 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.368706 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:09.368808 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.368783 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:09.368846 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.368823 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:09.444515 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.444478 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa2f0da-f709-4e28-a544-e7ff20f78633-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.444693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.444531 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-589zn\" (UniqueName: \"kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-kube-api-access-589zn\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.444693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.444585 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.444693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.444630 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.444693 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.444678 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.444911 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.444717 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.445430 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.445033 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.445430 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.445056 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.445430 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.445122 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.445430 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.445242 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa2f0da-f709-4e28-a544-e7ff20f78633-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.445430 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.445406 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.445751 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.445580 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.445810 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.445740 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.445912 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.445889 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.447284 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.447264 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.447690 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.447674 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.502222 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.502183 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-589zn\" (UniqueName: \"kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-kube-api-access-589zn\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.502444 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.502421 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:09.575362 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:09.575271 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:10.016317 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.016292 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc"] Nov 03 18:58:10.018417 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:58:10.018386 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa2f0da_f709_4e28_a544_e7ff20f78633.slice/crio-001e5fd832ee9e3003d0bdade7ce2533395d4798c0e390df7fe487e5fa961daf WatchSource:0}: Error finding container 001e5fd832ee9e3003d0bdade7ce2533395d4798c0e390df7fe487e5fa961daf: Status 404 returned error can't find the container with id 001e5fd832ee9e3003d0bdade7ce2533395d4798c0e390df7fe487e5fa961daf Nov 03 18:58:10.021473 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.021430 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:10.021563 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.021524 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:10.021622 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.021574 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:10.073672 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.073626 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" event={"ID":"8aa2f0da-f709-4e28-a544-e7ff20f78633","Type":"ContainerStarted","Data":"001e5fd832ee9e3003d0bdade7ce2533395d4798c0e390df7fe487e5fa961daf"} Nov 03 18:58:10.075448 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.075389 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" event={"ID":"107848ae-227c-4b41-ab04-19b284e22fd3","Type":"ContainerStarted","Data":"a5780b7a2c8d2df3fc3eb7a6bf06d015194703070e816248c08274cc6c59c6fa"} Nov 03 18:58:10.075448 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.075434 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" event={"ID":"107848ae-227c-4b41-ab04-19b284e22fd3","Type":"ContainerStarted","Data":"d0e346b6b27d1e4a9fe06ed0a3723fc625c530145ff206956986bc3d03d45974"} Nov 03 18:58:10.189605 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.189554 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" podStartSLOduration=2.189539738 podStartE2EDuration="2.189539738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:10.18804315 +0000 UTC m=+824.160604332" watchObservedRunningTime="2026-11-03 18:58:10.189539738 +0000 UTC m=+824.162100914" Nov 03 18:58:10.244527 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.244496 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5"] Nov 03 18:58:10.248357 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.248336 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.297254 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.297129 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-eb6ef8-openshift-default-dockercfg-rxz8d\"" Nov 03 18:58:10.309398 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.309367 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5"] Nov 03 18:58:10.361948 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.361919 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.362108 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.361969 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.362108 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.362056 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b59638-64eb-4da2-93aa-aa8f37354323-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.362108 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.362095 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhm9z\" (UniqueName: \"kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-kube-api-access-zhm9z\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.362253 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.362147 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.362253 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.362168 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.362253 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.362190 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.362253 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.362208 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.362435 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.362267 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.465013 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.464977 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.465245 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.465046 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.465245 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.465095 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b59638-64eb-4da2-93aa-aa8f37354323-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.465245 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.465140 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhm9z\" (UniqueName: \"kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-kube-api-access-zhm9z\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.465245 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.465188 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.465245 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.465220 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.465596 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.465249 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.465596 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.465271 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.465596 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.465316 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.465596 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.465476 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.469677 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.466118 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.469677 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.466394 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b59638-64eb-4da2-93aa-aa8f37354323-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.469677 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.466702 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.469677 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.469157 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.469677 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.469381 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.482736 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.482439 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.523528 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.523499 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.523706 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.523636 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhm9z\" (UniqueName: \"kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-kube-api-access-zhm9z\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.562502 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.562446 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:10.584057 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.584034 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh"] Nov 03 18:58:10.588212 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.588191 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.651182 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.650559 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh"] Nov 03 18:58:10.651182 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.650949 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-92596a-openshift-default-dockercfg-zf4cq\"" Nov 03 18:58:10.768181 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.768147 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.768381 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.768219 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.768381 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.768320 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.768381 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.768378 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.768497 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.768453 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.768497 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.768485 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.768556 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.768507 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.768556 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.768547 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.768621 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.768591 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cls4g\" (UniqueName: \"kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-kube-api-access-cls4g\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.869668 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.869605 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.869668 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.869643 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.869840 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.869677 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.869840 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.869705 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.869840 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.869735 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.869979 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.869874 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.869979 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.869916 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cls4g\" (UniqueName: \"kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-kube-api-access-cls4g\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.869979 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.869973 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.870117 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.870059 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.870117 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.870076 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.870388 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.870323 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.870554 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.870526 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.870676 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.870578 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.870676 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.870600 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.871970 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.871947 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.872179 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.872162 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.934780 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.934729 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.935390 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.935362 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cls4g\" (UniqueName: \"kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-kube-api-access-cls4g\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:10.938774 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.938745 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:10.939123 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:58:10.939090 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b59638_64eb_4da2_93aa_aa8f37354323.slice/crio-665dda3f773949e952995ee5216259b5cbfe83e5bcb6ff2f70dad64f822e57c4 WatchSource:0}: Error finding container 665dda3f773949e952995ee5216259b5cbfe83e5bcb6ff2f70dad64f822e57c4: Status 404 returned error can't find the container with id 665dda3f773949e952995ee5216259b5cbfe83e5bcb6ff2f70dad64f822e57c4 Nov 03 18:58:10.940637 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.940614 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5"] Nov 03 18:58:10.941948 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.941913 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:10.942047 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.941980 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:10.942047 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.942008 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:10.944721 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:10.944699 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:11.083635 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.083597 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" event={"ID":"a5b59638-64eb-4da2-93aa-aa8f37354323","Type":"ContainerStarted","Data":"9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4"} Nov 03 18:58:11.084103 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.083655 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" event={"ID":"a5b59638-64eb-4da2-93aa-aa8f37354323","Type":"ContainerStarted","Data":"665dda3f773949e952995ee5216259b5cbfe83e5bcb6ff2f70dad64f822e57c4"} Nov 03 18:58:11.085499 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.085466 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" event={"ID":"8aa2f0da-f709-4e28-a544-e7ff20f78633","Type":"ContainerStarted","Data":"20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70"} Nov 03 18:58:11.085705 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.085685 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:11.086627 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.086611 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:11.198370 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.198300 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" podStartSLOduration=1.198287861 podStartE2EDuration="1.198287861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:11.196725969 +0000 UTC m=+825.169287155" watchObservedRunningTime="2026-11-03 18:58:11.198287861 +0000 UTC m=+825.170849042" Nov 03 18:58:11.204298 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.204276 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:11.312526 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.312024 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" podStartSLOduration=2.312010086 podStartE2EDuration="2.312010086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:11.309903717 +0000 UTC m=+825.282464912" watchObservedRunningTime="2026-11-03 18:58:11.312010086 +0000 UTC m=+825.284571268" Nov 03 18:58:11.323193 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:58:11.323134 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:58:11.562839 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.562751 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:11.576399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.576365 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:11.581512 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.581487 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh"] Nov 03 18:58:11.583076 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.583055 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:11.584766 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:58:11.584735 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5455b012_cb7c_4ec5_bdc4_076bb20f9eae.slice/crio-75e48f5965ff8e14f317ed81ee898f89b13acc25c7d43161407ce8de77955113 WatchSource:0}: Error finding container 75e48f5965ff8e14f317ed81ee898f89b13acc25c7d43161407ce8de77955113: Status 404 returned error can't find the container with id 75e48f5965ff8e14f317ed81ee898f89b13acc25c7d43161407ce8de77955113 Nov 03 18:58:11.586682 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.586643 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:11.586763 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.586724 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:11.586802 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:11.586767 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:12.092905 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:12.092869 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" event={"ID":"5455b012-cb7c-4ec5-bdc4-076bb20f9eae","Type":"ContainerStarted","Data":"e24ec0ec767d393805c70f697a07daafbf651d3ceba903918f0549c90593b7b1"} Nov 03 18:58:12.093320 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:12.092932 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" event={"ID":"5455b012-cb7c-4ec5-bdc4-076bb20f9eae","Type":"ContainerStarted","Data":"75e48f5965ff8e14f317ed81ee898f89b13acc25c7d43161407ce8de77955113"} Nov 03 18:58:12.093407 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:12.093358 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:12.094117 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:12.094089 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:58:12.205289 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:12.205254 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:12.207235 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:12.207192 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" podStartSLOduration=2.207178012 podStartE2EDuration="2.207178012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:12.20623853 +0000 UTC m=+826.178799712" watchObservedRunningTime="2026-11-03 18:58:12.207178012 +0000 UTC m=+826.179739193" Nov 03 18:58:12.211263 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:12.211242 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:12.569264 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:12.569234 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:13.098398 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:13.098364 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:13.098820 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:13.098415 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:13.098918 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:13.098901 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:13.099027 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:13.099012 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:58:19.069491 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.069456 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2"] Nov 03 18:58:19.074602 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.074578 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.123274 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.123244 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-98860e-serving-cert\"" Nov 03 18:58:19.154804 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.154769 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg555\" (UniqueName: \"kubernetes.io/projected/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-kube-api-access-gg555\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.154954 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.154827 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-metrics-certs\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.155100 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.155083 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.155133 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.155109 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.172268 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.172230 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2"] Nov 03 18:58:19.255915 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.255874 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.255915 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.255908 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.256165 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.255940 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg555\" (UniqueName: \"kubernetes.io/projected/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-kube-api-access-gg555\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.256165 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.255975 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-metrics-certs\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.256165 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:58:19.256031 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-serving-cert: secret "maas-api-e2e-worker-w4-98860e-serving-cert" not found Nov 03 18:58:19.256165 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:58:19.256108 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-maas-api-tls podName:4410c3a0-9ebe-4f0b-88e8-45f2625284c7 nodeName:}" failed. No retries permitted until 2026-11-03 18:58:19.756084348 +0000 UTC m=+833.728645511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-maas-api-tls") pod "maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" (UID: "4410c3a0-9ebe-4f0b-88e8-45f2625284c7") : secret "maas-api-e2e-worker-w4-98860e-serving-cert" not found Nov 03 18:58:19.256609 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.256588 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.258479 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.258457 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-metrics-certs\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.320117 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.319290 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg555\" (UniqueName: \"kubernetes.io/projected/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-kube-api-access-gg555\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.761640 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.761608 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.764017 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.763985 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:19.988654 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:19.988621 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:20.352698 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:20.352673 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2"] Nov 03 18:58:20.354790 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:58:20.354762 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4410c3a0_9ebe_4f0b_88e8_45f2625284c7.slice/crio-460cac34826daba73fb645d7787ab67ffa41648670d8808da10edce983049b9e WatchSource:0}: Error finding container 460cac34826daba73fb645d7787ab67ffa41648670d8808da10edce983049b9e: Status 404 returned error can't find the container with id 460cac34826daba73fb645d7787ab67ffa41648670d8808da10edce983049b9e Nov 03 18:58:21.152688 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:21.152650 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" event={"ID":"4410c3a0-9ebe-4f0b-88e8-45f2625284c7","Type":"ContainerStarted","Data":"460cac34826daba73fb645d7787ab67ffa41648670d8808da10edce983049b9e"} Nov 03 18:58:22.159162 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:22.159127 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" event={"ID":"4410c3a0-9ebe-4f0b-88e8-45f2625284c7","Type":"ContainerStarted","Data":"19939e781b5ced82fa6558d02f6bf1664c7c788b61bf83ebe9d071c0bc9a2fdc"} Nov 03 18:58:22.159573 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:22.159193 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:22.271069 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:22.271012 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" podStartSLOduration=2.24818783 podStartE2EDuration="3.270999074s" totalImagesPullingTime="1.022811244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:22.26916756 +0000 UTC m=+836.241728742" watchObservedRunningTime="2026-11-03 18:58:22.270999074 +0000 UTC m=+836.243560255" Nov 03 18:58:23.932299 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:23.932269 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j"] Nov 03 18:58:23.936536 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:23.936501 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:23.985046 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:23.985024 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-fa2878-serving-cert\"" Nov 03 18:58:23.994503 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:23.994479 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j"] Nov 03 18:58:24.008791 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.008762 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7d428b-6291-4714-8848-5ada8ce9f341-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.008932 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.008835 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-metrics-certs\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.008932 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.008894 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.009053 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.008960 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2jv4\" (UniqueName: \"kubernetes.io/projected/5f7d428b-6291-4714-8848-5ada8ce9f341-kube-api-access-f2jv4\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.109483 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.109447 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.109645 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.109492 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2jv4\" (UniqueName: \"kubernetes.io/projected/5f7d428b-6291-4714-8848-5ada8ce9f341-kube-api-access-f2jv4\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.109645 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.109569 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7d428b-6291-4714-8848-5ada8ce9f341-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.109645 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.109589 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-metrics-certs\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.109645 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:58:24.109604 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-serving-cert: secret "maas-api-e2e-worker-w3-fa2878-serving-cert" not found Nov 03 18:58:24.109821 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:58:24.109687 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-maas-api-tls podName:5f7d428b-6291-4714-8848-5ada8ce9f341 nodeName:}" failed. No retries permitted until 2026-11-03 18:58:24.609667326 +0000 UTC m=+838.582228505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-maas-api-tls") pod "maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" (UID: "5f7d428b-6291-4714-8848-5ada8ce9f341") : secret "maas-api-e2e-worker-w3-fa2878-serving-cert" not found Nov 03 18:58:24.110322 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.110298 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7d428b-6291-4714-8848-5ada8ce9f341-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.112029 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.112007 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-metrics-certs\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.167296 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.167257 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2jv4\" (UniqueName: \"kubernetes.io/projected/5f7d428b-6291-4714-8848-5ada8ce9f341-kube-api-access-f2jv4\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.614230 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.614195 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.616727 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.616699 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:24.850273 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:24.850237 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:25.218024 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:25.217994 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j"] Nov 03 18:58:25.219553 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:58:25.219524 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7d428b_6291_4714_8848_5ada8ce9f341.slice/crio-7a67a10d0cc91aad928e5255c349a7d591b2f9b7c7e52e6c5dd56a8529ef4d66 WatchSource:0}: Error finding container 7a67a10d0cc91aad928e5255c349a7d591b2f9b7c7e52e6c5dd56a8529ef4d66: Status 404 returned error can't find the container with id 7a67a10d0cc91aad928e5255c349a7d591b2f9b7c7e52e6c5dd56a8529ef4d66 Nov 03 18:58:26.191654 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:26.191606 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" event={"ID":"5f7d428b-6291-4714-8848-5ada8ce9f341","Type":"ContainerStarted","Data":"7a67a10d0cc91aad928e5255c349a7d591b2f9b7c7e52e6c5dd56a8529ef4d66"} Nov 03 18:58:27.200679 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:27.200634 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" event={"ID":"5f7d428b-6291-4714-8848-5ada8ce9f341","Type":"ContainerStarted","Data":"af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2"} Nov 03 18:58:27.201065 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:27.200775 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:27.311124 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:27.311070 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" podStartSLOduration=3.390391196 podStartE2EDuration="4.3110578s" totalImagesPullingTime="920.666604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:27.310788875 +0000 UTC m=+841.283350060" watchObservedRunningTime="2026-11-03 18:58:27.3110578 +0000 UTC m=+841.283618982" Nov 03 18:58:28.171160 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:28.171128 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 18:58:28.924020 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:28.923444 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg"] Nov 03 18:58:28.930308 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:28.930279 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:28.983267 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:28.983238 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg"] Nov 03 18:58:28.983452 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:28.983296 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-619731-serving-cert\"" Nov 03 18:58:29.084262 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.084223 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.084262 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.084268 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtkf\" (UniqueName: \"kubernetes.io/projected/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-kube-api-access-7jtkf\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.084508 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.084440 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-metrics-certs\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.084557 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.084518 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.193090 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.192447 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.193090 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.192497 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtkf\" (UniqueName: \"kubernetes.io/projected/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-kube-api-access-7jtkf\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.193090 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.192557 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-metrics-certs\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.193090 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.192652 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.194379 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.193608 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.194379 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:58:29.193724 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-serving-cert: secret "maas-api-e2e-worker-w2-619731-serving-cert" not found Nov 03 18:58:29.194379 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:58:29.193805 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-maas-api-tls podName:c214f760-6b74-4f5a-bf17-5dc2fe17ed7f nodeName:}" failed. No retries permitted until 2026-11-03 18:58:29.693780088 +0000 UTC m=+843.666341288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-maas-api-tls") pod "maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" (UID: "c214f760-6b74-4f5a-bf17-5dc2fe17ed7f") : secret "maas-api-e2e-worker-w2-619731-serving-cert" not found Nov 03 18:58:29.200237 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.200202 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-metrics-certs\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.253587 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.253557 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtkf\" (UniqueName: \"kubernetes.io/projected/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-kube-api-access-7jtkf\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.697045 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.696998 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.699437 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.699414 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:29.843500 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:29.843466 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:30.210459 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:30.210438 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg"] Nov 03 18:58:30.212163 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:58:30.212126 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc214f760_6b74_4f5a_bf17_5dc2fe17ed7f.slice/crio-dc6c8262960cfd5f3c7a7c88b36f5659f9534a7d2765943aa66391666f31e204 WatchSource:0}: Error finding container dc6c8262960cfd5f3c7a7c88b36f5659f9534a7d2765943aa66391666f31e204: Status 404 returned error can't find the container with id dc6c8262960cfd5f3c7a7c88b36f5659f9534a7d2765943aa66391666f31e204 Nov 03 18:58:31.265751 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:31.264629 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" event={"ID":"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f","Type":"ContainerStarted","Data":"9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a"} Nov 03 18:58:31.265751 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:31.264701 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" event={"ID":"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f","Type":"ContainerStarted","Data":"dc6c8262960cfd5f3c7a7c88b36f5659f9534a7d2765943aa66391666f31e204"} Nov 03 18:58:31.265751 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:31.265693 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:31.379453 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:31.379336 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" podStartSLOduration=2.574190795 podStartE2EDuration="3.379309254s" totalImagesPullingTime="805.118459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:31.377384263 +0000 UTC m=+845.349945446" watchObservedRunningTime="2026-11-03 18:58:31.379309254 +0000 UTC m=+845.351870458" Nov 03 18:58:33.211690 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:33.211662 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 18:58:33.923493 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:33.923459 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw"] Nov 03 18:58:33.927565 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:33.927546 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:33.952081 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:33.952058 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-metrics-certs\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:33.952207 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:33.952122 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-maas-api-tls\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:33.952253 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:33.952230 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv84d\" (UniqueName: \"kubernetes.io/projected/4d5e2740-ab79-4f6e-97cb-bac3339fde21-kube-api-access-dv84d\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:33.952291 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:33.952267 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5e2740-ab79-4f6e-97cb-bac3339fde21-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:34.019304 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:34.019270 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-010ed3-serving-cert\"" Nov 03 18:58:34.025167 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:34.025143 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw"] Nov 03 18:58:34.053489 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:34.053461 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv84d\" (UniqueName: \"kubernetes.io/projected/4d5e2740-ab79-4f6e-97cb-bac3339fde21-kube-api-access-dv84d\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:34.053616 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:34.053502 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5e2740-ab79-4f6e-97cb-bac3339fde21-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:34.053616 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:34.053549 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-metrics-certs\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:34.053729 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:34.053614 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-maas-api-tls\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:34.053729 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:58:34.053713 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-serving-cert: secret "maas-api-e2e-worker-w1-010ed3-serving-cert" not found Nov 03 18:58:34.053833 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:58:34.053770 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-maas-api-tls podName:4d5e2740-ab79-4f6e-97cb-bac3339fde21 nodeName:}" failed. No retries permitted until 2026-11-03 18:58:34.553754413 +0000 UTC m=+848.526315581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-maas-api-tls") pod "maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" (UID: "4d5e2740-ab79-4f6e-97cb-bac3339fde21") : secret "maas-api-e2e-worker-w1-010ed3-serving-cert" not found Nov 03 18:58:34.054281 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:34.054262 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5e2740-ab79-4f6e-97cb-bac3339fde21-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:34.055994 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:34.055976 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-metrics-certs\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:34.120135 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:34.119944 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv84d\" (UniqueName: \"kubernetes.io/projected/4d5e2740-ab79-4f6e-97cb-bac3339fde21-kube-api-access-dv84d\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:34.560156 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:34.560124 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-maas-api-tls\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:34.562532 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:34.562507 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-maas-api-tls\") pod \"maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:34.841170 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:34.841098 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:35.210174 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:35.210149 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw"] Nov 03 18:58:35.211588 ip-10-0-139-179 kubenswrapper[5019]: W1103 18:58:35.211560 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5e2740_ab79_4f6e_97cb_bac3339fde21.slice/crio-89bae917f1f51fa48532c50dd53abad4b2e5e43790eea5bff94c1eb6e9e2a4a8 WatchSource:0}: Error finding container 89bae917f1f51fa48532c50dd53abad4b2e5e43790eea5bff94c1eb6e9e2a4a8: Status 404 returned error can't find the container with id 89bae917f1f51fa48532c50dd53abad4b2e5e43790eea5bff94c1eb6e9e2a4a8 Nov 03 18:58:35.286125 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:35.286088 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" event={"ID":"4d5e2740-ab79-4f6e-97cb-bac3339fde21","Type":"ContainerStarted","Data":"89bae917f1f51fa48532c50dd53abad4b2e5e43790eea5bff94c1eb6e9e2a4a8"} Nov 03 18:58:36.294021 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:36.293981 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" event={"ID":"4d5e2740-ab79-4f6e-97cb-bac3339fde21","Type":"ContainerStarted","Data":"11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29"} Nov 03 18:58:36.294500 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:36.294064 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:36.408052 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:36.407946 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" podStartSLOduration=2.570095937 podStartE2EDuration="3.407932217s" totalImagesPullingTime="837.83628ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:36.406251355 +0000 UTC m=+850.378812520" watchObservedRunningTime="2026-11-03 18:58:36.407932217 +0000 UTC m=+850.380493402" Nov 03 18:58:38.289430 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:38.289398 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 18:58:41.317566 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:58:41.317525 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:58:42.305431 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:42.305389 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 18:58:46.352755 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:46.352675 5019 generic.go:356] "Generic (PLEG): container finished" podID="107848ae-227c-4b41-ab04-19b284e22fd3" containerID="a5780b7a2c8d2df3fc3eb7a6bf06d015194703070e816248c08274cc6c59c6fa" exitCode=0 Nov 03 18:58:46.353155 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:46.352741 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" event={"ID":"107848ae-227c-4b41-ab04-19b284e22fd3","Type":"ContainerDied","Data":"a5780b7a2c8d2df3fc3eb7a6bf06d015194703070e816248c08274cc6c59c6fa"} Nov 03 18:58:46.353155 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:46.353136 5019 scope.go:122] "RemoveContainer" containerID="a5780b7a2c8d2df3fc3eb7a6bf06d015194703070e816248c08274cc6c59c6fa" Nov 03 18:58:46.354439 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:46.354413 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:46.354505 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:46.354477 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:46.354543 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:46.354510 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:47.360264 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:47.360229 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" event={"ID":"107848ae-227c-4b41-ab04-19b284e22fd3","Type":"ContainerStarted","Data":"06befcbed3c89285f103138870074aa0e5d2e1ebb33b7144c0025e7b2880f3de"} Nov 03 18:58:47.939116 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:47.939083 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:47.939300 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:47.939229 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:58:47.939300 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:47.939275 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 18:58:48.938947 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:48.938895 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:58:48.939634 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:48.938991 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 18:58:49.938977 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:49.938939 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:58:49.939399 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:49.939005 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 18:58:50.939319 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:50.939279 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:58:50.939822 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:50.939380 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 18:58:51.939226 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:51.939193 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:58:51.939462 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:51.939281 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 18:58:52.943625 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:52.943595 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:52.944034 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:52.944012 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:52.944582 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:52.944565 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 18:58:57.459820 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:57.459789 5019 generic.go:356] "Generic (PLEG): container finished" podID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerID="9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4" exitCode=0 Nov 03 18:58:57.460394 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:57.459869 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" event={"ID":"a5b59638-64eb-4da2-93aa-aa8f37354323","Type":"ContainerDied","Data":"9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4"} Nov 03 18:58:57.460394 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:57.460380 5019 scope.go:122] "RemoveContainer" containerID="9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4" Nov 03 18:58:57.461897 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:57.461856 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:57.461994 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:57.461976 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:57.462041 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:57.462025 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:58:58.098851 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:58.098814 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:58.495420 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:58.495384 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" event={"ID":"a5b59638-64eb-4da2-93aa-aa8f37354323","Type":"ContainerStarted","Data":"d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729"} Nov 03 18:58:58.580346 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:58.571016 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:58:58.580346 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:58.571141 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 18:58:58.580346 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:58.571187 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 18:58:59.581591 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:59.577135 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 18:58:59.581591 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:58:59.577194 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 18:59:00.570457 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:00.570422 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 18:59:00.570746 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:00.570712 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 18:59:01.581352 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:01.580383 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 18:59:01.581352 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:01.580479 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 18:59:02.567713 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:02.567679 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 18:59:02.568455 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:02.568422 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 18:59:03.581485 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:03.581440 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 18:59:03.582104 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:03.582070 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 18:59:04.586687 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:04.586650 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:59:04.588717 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:04.588694 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:59:04.602218 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:04.602187 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 18:59:08.564577 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:08.564491 5019 generic.go:356] "Generic (PLEG): container finished" podID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerID="20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70" exitCode=0 Nov 03 18:59:08.564577 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:08.564563 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" event={"ID":"8aa2f0da-f709-4e28-a544-e7ff20f78633","Type":"ContainerDied","Data":"20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70"} Nov 03 18:59:08.565113 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:08.564959 5019 scope.go:122] "RemoveContainer" containerID="20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70" Nov 03 18:59:08.566261 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:08.566223 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:59:08.566322 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:08.566298 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:59:08.566434 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:08.566355 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:59:09.626381 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:09.626311 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" event={"ID":"8aa2f0da-f709-4e28-a544-e7ff20f78633","Type":"ContainerStarted","Data":"a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6"} Nov 03 18:59:10.581349 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:10.580928 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:59:10.587273 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:10.587233 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 18:59:10.587528 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:10.587503 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 18:59:11.341552 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:59:11.341516 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 18:59:11.580230 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:11.579780 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 18:59:11.580230 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:11.579843 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 18:59:12.100352 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:12.096433 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:59:12.581350 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:12.578724 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 18:59:12.581350 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:12.578803 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 18:59:13.588349 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:13.587730 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 18:59:13.588349 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:13.587809 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 18:59:14.577743 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:14.577702 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 18:59:14.578055 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:14.578024 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 18:59:15.580394 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:15.580357 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:59:15.581545 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:15.581520 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 18:59:18.683603 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:18.683573 5019 generic.go:356] "Generic (PLEG): container finished" podID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerID="e24ec0ec767d393805c70f697a07daafbf651d3ceba903918f0549c90593b7b1" exitCode=0 Nov 03 18:59:18.684005 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:18.683646 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" event={"ID":"5455b012-cb7c-4ec5-bdc4-076bb20f9eae","Type":"ContainerDied","Data":"e24ec0ec767d393805c70f697a07daafbf651d3ceba903918f0549c90593b7b1"} Nov 03 18:59:18.684047 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:18.684018 5019 scope.go:122] "RemoveContainer" containerID="e24ec0ec767d393805c70f697a07daafbf651d3ceba903918f0549c90593b7b1" Nov 03 18:59:18.685282 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:18.685254 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:59:18.685370 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:18.685351 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:59:18.685414 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:18.685386 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 18:59:19.694516 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:19.694476 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" event={"ID":"5455b012-cb7c-4ec5-bdc4-076bb20f9eae","Type":"ContainerStarted","Data":"b38381705dfb80664400ebde74d7a72406aaa3fa89cb394395069f0bfd3829f3"} Nov 03 18:59:20.205228 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:20.205200 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:59:20.205539 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:20.205521 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 18:59:20.205590 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:20.205561 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 18:59:21.205501 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:21.205465 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 18:59:21.205925 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:21.205519 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 18:59:22.204712 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:22.204674 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 18:59:22.204943 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:22.204751 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 18:59:23.205620 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:23.205574 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 18:59:23.206047 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:23.205647 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 18:59:24.207536 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:24.207499 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 18:59:24.208217 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:24.208155 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 18:59:25.209393 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:25.209365 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:59:25.209776 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:25.209661 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:59:25.210282 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:25.210262 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 18:59:26.228246 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:26.228216 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 18:59:26.229529 ip-10-0-139-179 kubenswrapper[5019]: I1103 18:59:26.229508 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 18:59:41.299456 ip-10-0-139-179 kubenswrapper[5019]: E1103 18:59:41.299417 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:00:00.202472 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.202440 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264"] Nov 03 19:00:00.207444 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.207425 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" Nov 03 19:00:00.236656 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.236624 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sphb\" (UniqueName: \"kubernetes.io/projected/a07404fe-c011-4308-aacb-64a7f1e1b94c-kube-api-access-2sphb\") pod \"maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264\" (UID: \"a07404fe-c011-4308-aacb-64a7f1e1b94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" Nov 03 19:00:00.236820 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.236795 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07404fe-c011-4308-aacb-64a7f1e1b94c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264\" (UID: \"a07404fe-c011-4308-aacb-64a7f1e1b94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" Nov 03 19:00:00.281549 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.281517 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264"] Nov 03 19:00:00.332946 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.332913 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\"" Nov 03 19:00:00.338417 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.338386 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sphb\" (UniqueName: \"kubernetes.io/projected/a07404fe-c011-4308-aacb-64a7f1e1b94c-kube-api-access-2sphb\") pod \"maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264\" (UID: \"a07404fe-c011-4308-aacb-64a7f1e1b94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" Nov 03 19:00:00.338539 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.338479 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07404fe-c011-4308-aacb-64a7f1e1b94c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264\" (UID: \"a07404fe-c011-4308-aacb-64a7f1e1b94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" Nov 03 19:00:00.339130 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.339110 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07404fe-c011-4308-aacb-64a7f1e1b94c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264\" (UID: \"a07404fe-c011-4308-aacb-64a7f1e1b94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" Nov 03 19:00:00.395835 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.395798 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sphb\" (UniqueName: \"kubernetes.io/projected/a07404fe-c011-4308-aacb-64a7f1e1b94c-kube-api-access-2sphb\") pod \"maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264\" (UID: \"a07404fe-c011-4308-aacb-64a7f1e1b94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" Nov 03 19:00:00.520913 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.520839 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" Nov 03 19:00:00.889368 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.888761 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264"] Nov 03 19:00:00.894549 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:00:00.894513 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07404fe_c011_4308_aacb_64a7f1e1b94c.slice/crio-5301af8b189b35f7c6032628d151feb60f02eed95ba4a79b8defa9b17bdc8982 WatchSource:0}: Error finding container 5301af8b189b35f7c6032628d151feb60f02eed95ba4a79b8defa9b17bdc8982: Status 404 returned error can't find the container with id 5301af8b189b35f7c6032628d151feb60f02eed95ba4a79b8defa9b17bdc8982 Nov 03 19:00:00.952158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:00.952128 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" event={"ID":"a07404fe-c011-4308-aacb-64a7f1e1b94c","Type":"ContainerStarted","Data":"5301af8b189b35f7c6032628d151feb60f02eed95ba4a79b8defa9b17bdc8982"} Nov 03 19:00:09.003816 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:09.003780 5019 generic.go:356] "Generic (PLEG): container finished" podID="a07404fe-c011-4308-aacb-64a7f1e1b94c" containerID="39e017b4ea290dee969109740f21a8f6d18187d47dd6508d46f1b9d5dbd73732" exitCode=60 Nov 03 19:00:09.004192 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:09.003838 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" event={"ID":"a07404fe-c011-4308-aacb-64a7f1e1b94c","Type":"ContainerDied","Data":"39e017b4ea290dee969109740f21a8f6d18187d47dd6508d46f1b9d5dbd73732"} Nov 03 19:00:09.004192 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:09.004092 5019 scope.go:122] "RemoveContainer" containerID="39e017b4ea290dee969109740f21a8f6d18187d47dd6508d46f1b9d5dbd73732" Nov 03 19:00:10.012080 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:10.011994 5019 generic.go:356] "Generic (PLEG): container finished" podID="a07404fe-c011-4308-aacb-64a7f1e1b94c" containerID="2953f160f16f2ab9d29486477c38b0a15068431eb1ebf6f15aa112c81142fe7f" exitCode=60 Nov 03 19:00:10.012080 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:10.012049 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" event={"ID":"a07404fe-c011-4308-aacb-64a7f1e1b94c","Type":"ContainerDied","Data":"2953f160f16f2ab9d29486477c38b0a15068431eb1ebf6f15aa112c81142fe7f"} Nov 03 19:00:10.012646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:10.012094 5019 scope.go:122] "RemoveContainer" containerID="39e017b4ea290dee969109740f21a8f6d18187d47dd6508d46f1b9d5dbd73732" Nov 03 19:00:10.012646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:10.012353 5019 scope.go:122] "RemoveContainer" containerID="2953f160f16f2ab9d29486477c38b0a15068431eb1ebf6f15aa112c81142fe7f" Nov 03 19:00:10.012646 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:00:10.012593 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264_odh-ai-gateway-infra(a07404fe-c011-4308-aacb-64a7f1e1b94c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" Nov 03 19:00:11.019579 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:11.019550 5019 scope.go:122] "RemoveContainer" containerID="2953f160f16f2ab9d29486477c38b0a15068431eb1ebf6f15aa112c81142fe7f" Nov 03 19:00:11.019955 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:00:11.019826 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264_odh-ai-gateway-infra(a07404fe-c011-4308-aacb-64a7f1e1b94c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" Nov 03 19:00:11.297800 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:00:11.297723 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:00:23.143240 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:23.143208 5019 scope.go:122] "RemoveContainer" containerID="2953f160f16f2ab9d29486477c38b0a15068431eb1ebf6f15aa112c81142fe7f" Nov 03 19:00:24.097593 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:24.097553 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" event={"ID":"a07404fe-c011-4308-aacb-64a7f1e1b94c","Type":"ContainerStarted","Data":"802409c8791cf49e51c94cc60a51eae0ea941ffde7e028e2ba7e7f2e166c1254"} Nov 03 19:00:24.205996 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:24.205948 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" podStartSLOduration=17.303362696 podStartE2EDuration="24.205934619s" totalImagesPullingTime="6.902571923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:24.205129355 +0000 UTC m=+958.177690536" watchObservedRunningTime="2026-11-03 19:00:24.205934619 +0000 UTC m=+958.178495801" Nov 03 19:00:25.105438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:25.105393 5019 generic.go:356] "Generic (PLEG): container finished" podID="a07404fe-c011-4308-aacb-64a7f1e1b94c" containerID="802409c8791cf49e51c94cc60a51eae0ea941ffde7e028e2ba7e7f2e166c1254" exitCode=60 Nov 03 19:00:25.105644 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:25.105470 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" event={"ID":"a07404fe-c011-4308-aacb-64a7f1e1b94c","Type":"ContainerDied","Data":"802409c8791cf49e51c94cc60a51eae0ea941ffde7e028e2ba7e7f2e166c1254"} Nov 03 19:00:25.105644 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:25.105546 5019 scope.go:122] "RemoveContainer" containerID="2953f160f16f2ab9d29486477c38b0a15068431eb1ebf6f15aa112c81142fe7f" Nov 03 19:00:25.105957 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:25.105942 5019 scope.go:122] "RemoveContainer" containerID="802409c8791cf49e51c94cc60a51eae0ea941ffde7e028e2ba7e7f2e166c1254" Nov 03 19:00:25.106231 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:00:25.106209 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264_odh-ai-gateway-infra(a07404fe-c011-4308-aacb-64a7f1e1b94c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" Nov 03 19:00:25.214748 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:25.214715 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264"] Nov 03 19:00:26.275337 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:26.275298 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" Nov 03 19:00:26.324838 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:26.324803 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07404fe-c011-4308-aacb-64a7f1e1b94c-kube-api-access-2sphb\" (UniqueName: \"kubernetes.io/projected/a07404fe-c011-4308-aacb-64a7f1e1b94c-kube-api-access-2sphb\") pod \"a07404fe-c011-4308-aacb-64a7f1e1b94c\" (UID: \"a07404fe-c011-4308-aacb-64a7f1e1b94c\") " Nov 03 19:00:26.326912 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:26.326888 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07404fe-c011-4308-aacb-64a7f1e1b94c-kube-api-access-2sphb" pod "a07404fe-c011-4308-aacb-64a7f1e1b94c" (UID: "a07404fe-c011-4308-aacb-64a7f1e1b94c"). InnerVolumeSpecName "kube-api-access-2sphb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:26.425890 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:26.425853 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07404fe-c011-4308-aacb-64a7f1e1b94c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07404fe-c011-4308-aacb-64a7f1e1b94c-maas-api-service-ca\") pod \"a07404fe-c011-4308-aacb-64a7f1e1b94c\" (UID: \"a07404fe-c011-4308-aacb-64a7f1e1b94c\") " Nov 03 19:00:26.426263 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:26.426235 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07404fe-c011-4308-aacb-64a7f1e1b94c-maas-api-service-ca" pod "a07404fe-c011-4308-aacb-64a7f1e1b94c" (UID: "a07404fe-c011-4308-aacb-64a7f1e1b94c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:26.426407 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:26.426388 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sphb\" (UniqueName: \"kubernetes.io/projected/a07404fe-c011-4308-aacb-64a7f1e1b94c-kube-api-access-2sphb\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:26.426457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:26.426413 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07404fe-c011-4308-aacb-64a7f1e1b94c-maas-api-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:27.121131 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:27.121098 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" event={"ID":"a07404fe-c011-4308-aacb-64a7f1e1b94c","Type":"ContainerDied","Data":"5301af8b189b35f7c6032628d151feb60f02eed95ba4a79b8defa9b17bdc8982"} Nov 03 19:00:27.121316 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:27.121137 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264" Nov 03 19:00:27.121316 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:27.121160 5019 scope.go:122] "RemoveContainer" containerID="802409c8791cf49e51c94cc60a51eae0ea941ffde7e028e2ba7e7f2e166c1254" Nov 03 19:00:27.275169 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:27.275137 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264"] Nov 03 19:00:27.279289 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:27.279263 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895540-gh264"] Nov 03 19:00:28.149136 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:00:28.149094 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" path="/var/lib/kubelet/pods/a07404fe-c011-4308-aacb-64a7f1e1b94c/volumes" Nov 03 19:00:41.296956 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:00:41.296916 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:01:11.298991 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:01:11.298955 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:01:25.002602 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.002571 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl"] Nov 03 19:01:25.003106 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.003092 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" containerName="cleanup" Nov 03 19:01:25.003151 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.003109 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" containerName="cleanup" Nov 03 19:01:25.003151 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.003133 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" containerName="cleanup" Nov 03 19:01:25.003151 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.003139 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" containerName="cleanup" Nov 03 19:01:25.003151 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.003147 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" containerName="cleanup" Nov 03 19:01:25.003151 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.003152 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" containerName="cleanup" Nov 03 19:01:25.003299 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.003259 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" containerName="cleanup" Nov 03 19:01:25.003299 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.003273 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" containerName="cleanup" Nov 03 19:01:25.003299 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.003280 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07404fe-c011-4308-aacb-64a7f1e1b94c" containerName="cleanup" Nov 03 19:01:25.006732 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.006714 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.102479 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.102446 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-eb6ef8\"/\"kube-root-ca.crt\"" Nov 03 19:01:25.102652 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.102465 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-eb6ef8\"/\"default-dockercfg-pzhxj\"" Nov 03 19:01:25.102652 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.102471 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-eb6ef8\"/\"premium-w6-eb6ef8-kserve-self-signed-certs\"" Nov 03 19:01:25.102652 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.102544 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-eb6ef8\"/\"openshift-service-ca.crt\"" Nov 03 19:01:25.105031 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.105006 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl"] Nov 03 19:01:25.154958 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.154927 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-dshm\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.155142 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.155101 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tls-certs\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.155207 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.155171 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-home\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.155261 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.155202 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j26n2\" (UniqueName: \"kubernetes.io/projected/6b13be41-3746-4fc5-b7a5-2a2be4031df0-kube-api-access-j26n2\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.155340 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.155305 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-model-cache\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.155396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.155381 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tmp-dir\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.279441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.257910 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tls-certs\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.279441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.258006 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-home\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.279441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.258055 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j26n2\" (UniqueName: \"kubernetes.io/projected/6b13be41-3746-4fc5-b7a5-2a2be4031df0-kube-api-access-j26n2\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.279441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.258102 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-model-cache\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.279441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.258143 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tmp-dir\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.279441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.258198 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-dshm\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.279441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.270878 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tls-certs\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.279441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.271168 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-home\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.279441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.271490 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-model-cache\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.279441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.272904 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tmp-dir\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.287617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.287541 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-dshm\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.327068 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.327028 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j26n2\" (UniqueName: \"kubernetes.io/projected/6b13be41-3746-4fc5-b7a5-2a2be4031df0-kube-api-access-j26n2\") pod \"premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.336898 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.336872 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:25.493651 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.493618 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl"] Nov 03 19:01:25.498914 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.498890 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.548238 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.548148 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-98860e-openshift-default-dockercfg-p56jx\"" Nov 03 19:01:25.557656 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.557628 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl"] Nov 03 19:01:25.561744 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.561716 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.561880 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.561748 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35207ff1-165f-4250-8315-7938cbf38842-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.561880 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.561768 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsqdt\" (UniqueName: \"kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-kube-api-access-rsqdt\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.561992 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.561950 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.561992 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.561973 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.562115 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.562100 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.562166 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.562127 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.562222 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.562162 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.562269 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.562214 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35207ff1-165f-4250-8315-7938cbf38842-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.663777 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.663744 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.663973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.663789 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.663973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.663822 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.663973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.663838 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35207ff1-165f-4250-8315-7938cbf38842-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.663973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.663879 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.663973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.663955 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35207ff1-165f-4250-8315-7938cbf38842-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.664207 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.663992 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsqdt\" (UniqueName: \"kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-kube-api-access-rsqdt\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.664207 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.664086 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.664207 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.664118 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.664593 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.664564 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.664593 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.664587 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.664769 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.664684 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.664769 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.664746 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.665043 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.665020 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35207ff1-165f-4250-8315-7938cbf38842-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.666532 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.666515 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.666774 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.666756 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35207ff1-165f-4250-8315-7938cbf38842-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.723871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.723839 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.724045 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.724027 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsqdt\" (UniqueName: \"kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-kube-api-access-rsqdt\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:25.745257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.745235 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl"] Nov 03 19:01:25.747067 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:01:25.747034 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b13be41_3746_4fc5_b7a5_2a2be4031df0.slice/crio-5db34208a06193bfba6c01ecd6257de4cda3e6f0c7e9f6d960709ffffc0bb1c6 WatchSource:0}: Error finding container 5db34208a06193bfba6c01ecd6257de4cda3e6f0c7e9f6d960709ffffc0bb1c6: Status 404 returned error can't find the container with id 5db34208a06193bfba6c01ecd6257de4cda3e6f0c7e9f6d960709ffffc0bb1c6 Nov 03 19:01:25.815877 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:25.815840 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:26.240319 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:26.240236 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl"] Nov 03 19:01:26.242797 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:01:26.242764 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35207ff1_165f_4250_8315_7938cbf38842.slice/crio-77abc2b593b30697cd3e714bf32305d56f4fe9acd66bbf8bf3d0c210f390bdb5 WatchSource:0}: Error finding container 77abc2b593b30697cd3e714bf32305d56f4fe9acd66bbf8bf3d0c210f390bdb5: Status 404 returned error can't find the container with id 77abc2b593b30697cd3e714bf32305d56f4fe9acd66bbf8bf3d0c210f390bdb5 Nov 03 19:01:26.245085 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:26.245049 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:01:26.245181 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:26.245117 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:01:26.245181 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:26.245147 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:01:26.462202 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:26.462155 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" event={"ID":"35207ff1-165f-4250-8315-7938cbf38842","Type":"ContainerStarted","Data":"b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1"} Nov 03 19:01:26.462421 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:26.462228 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" event={"ID":"35207ff1-165f-4250-8315-7938cbf38842","Type":"ContainerStarted","Data":"77abc2b593b30697cd3e714bf32305d56f4fe9acd66bbf8bf3d0c210f390bdb5"} Nov 03 19:01:26.463678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:26.463653 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" event={"ID":"6b13be41-3746-4fc5-b7a5-2a2be4031df0","Type":"ContainerStarted","Data":"244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e"} Nov 03 19:01:26.463787 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:26.463688 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" event={"ID":"6b13be41-3746-4fc5-b7a5-2a2be4031df0","Type":"ContainerStarted","Data":"5db34208a06193bfba6c01ecd6257de4cda3e6f0c7e9f6d960709ffffc0bb1c6"} Nov 03 19:01:26.590181 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:26.590078 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" podStartSLOduration=1.590064723 podStartE2EDuration="1.590064723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:26.587632416 +0000 UTC m=+1020.560193598" watchObservedRunningTime="2026-11-03 19:01:26.590064723 +0000 UTC m=+1020.562625899" Nov 03 19:01:26.697992 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:26.697934 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podStartSLOduration=2.697917381 podStartE2EDuration="2.697917381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:26.696029715 +0000 UTC m=+1020.668590897" watchObservedRunningTime="2026-11-03 19:01:26.697917381 +0000 UTC m=+1020.670478564" Nov 03 19:01:26.820536 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:26.819309 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:27.337099 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:27.337066 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:27.818285 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:27.818246 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:01:27.818512 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:27.818296 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" podUID="35207ff1-165f-4250-8315-7938cbf38842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 19:01:28.816237 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:28.816193 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:01:28.816668 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:28.816257 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" podUID="35207ff1-165f-4250-8315-7938cbf38842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 19:01:29.816362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:29.816306 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:01:29.816764 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:29.816407 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" podUID="35207ff1-165f-4250-8315-7938cbf38842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 19:01:30.817130 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:30.817098 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:01:30.817571 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:30.817158 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" podUID="35207ff1-165f-4250-8315-7938cbf38842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 19:01:31.816350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:31.816290 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:01:31.816521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:31.816375 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" podUID="35207ff1-165f-4250-8315-7938cbf38842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 19:01:32.912228 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:32.912188 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:32.912759 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:32.912733 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:32.913478 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:32.913445 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:01:35.337794 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:35.337762 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:35.349988 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:35.349966 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:35.558286 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:01:35.558261 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:01:41.298437 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:01:41.298396 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:02:10.851322 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:10.851291 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9"] Nov 03 19:02:10.856507 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:10.856488 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:10.922233 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:10.922196 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:10.922419 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:10.922249 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b903d93f-7b45-486c-8879-7a9843c324ae-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:10.922419 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:10.922355 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b903d93f-7b45-486c-8879-7a9843c324ae-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:10.922538 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:10.922498 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:10.922602 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:10.922580 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:10.922640 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:10.922621 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:10.922684 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:10.922653 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:10.922742 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:10.922731 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mm49\" (UniqueName: \"kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-kube-api-access-7mm49\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:10.922779 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:10.922755 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:10.955918 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:10.955886 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9"] Nov 03 19:02:11.023661 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.023628 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.023822 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.023683 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mm49\" (UniqueName: \"kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-kube-api-access-7mm49\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.023822 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.023705 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.023822 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.023724 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.023822 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.023746 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b903d93f-7b45-486c-8879-7a9843c324ae-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.023822 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.023773 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b903d93f-7b45-486c-8879-7a9843c324ae-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.024054 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.023863 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.024054 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.023936 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.024054 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.023990 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.024212 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.024154 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.024278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.024201 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.024379 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.024355 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.024453 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.024413 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.024647 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.024627 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b903d93f-7b45-486c-8879-7a9843c324ae-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.026059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.026036 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.026178 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.026160 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b903d93f-7b45-486c-8879-7a9843c324ae-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.081301 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.081277 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mm49\" (UniqueName: \"kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-kube-api-access-7mm49\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.081301 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.081294 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.169553 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.169523 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:11.308827 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:02:11.308792 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:02:11.454602 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.454533 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw"] Nov 03 19:02:11.459642 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.459621 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.508737 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.508705 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-619731-openshift-default-dockercfg-m7qhx\"" Nov 03 19:02:11.520459 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.520438 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw"] Nov 03 19:02:11.529620 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.529586 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.529743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.529639 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.529743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.529725 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d614c12-4321-4857-a576-9552dc037c70-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.529848 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.529747 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5vps\" (UniqueName: \"kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-kube-api-access-l5vps\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.529848 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.529780 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d614c12-4321-4857-a576-9552dc037c70-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.529954 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.529882 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.530014 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.529946 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.530069 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.530041 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.530169 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.530145 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.610552 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.610527 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9"] Nov 03 19:02:11.612682 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:02:11.612646 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb903d93f_7b45_486c_8879_7a9843c324ae.slice/crio-3080063c9627af94d76953eb7de6c531e71e968e918c9f7316e28dc8888719f9 WatchSource:0}: Error finding container 3080063c9627af94d76953eb7de6c531e71e968e918c9f7316e28dc8888719f9: Status 404 returned error can't find the container with id 3080063c9627af94d76953eb7de6c531e71e968e918c9f7316e28dc8888719f9 Nov 03 19:02:11.615210 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.615178 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:02:11.615278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.615245 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:02:11.615314 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.615275 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:02:11.631389 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.631305 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.631499 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.631418 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.631499 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.631451 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.631499 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.631474 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d614c12-4321-4857-a576-9552dc037c70-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.631657 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.631504 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5vps\" (UniqueName: \"kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-kube-api-access-l5vps\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.631657 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.631541 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d614c12-4321-4857-a576-9552dc037c70-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.631657 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.631583 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.631657 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.631613 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.631862 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.631848 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.631952 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.631931 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.632254 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.631963 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.632512 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.632273 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d614c12-4321-4857-a576-9552dc037c70-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.632706 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.632293 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.632929 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.632909 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.634441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.634418 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d614c12-4321-4857-a576-9552dc037c70-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.634540 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.634483 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.688769 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.688738 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.689117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.689098 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5vps\" (UniqueName: \"kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-kube-api-access-l5vps\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.774587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.774493 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:11.796747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.796711 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" event={"ID":"b903d93f-7b45-486c-8879-7a9843c324ae","Type":"ContainerStarted","Data":"c6e79d61c3fad0bd462c60647b1eacee049486c589b829dd0327732917ee3174"} Nov 03 19:02:11.796897 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.796790 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" event={"ID":"b903d93f-7b45-486c-8879-7a9843c324ae","Type":"ContainerStarted","Data":"3080063c9627af94d76953eb7de6c531e71e968e918c9f7316e28dc8888719f9"} Nov 03 19:02:11.916582 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:11.916534 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" podStartSLOduration=1.916520078 podStartE2EDuration="1.916520078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:11.91450174 +0000 UTC m=+1065.887062922" watchObservedRunningTime="2026-11-03 19:02:11.916520078 +0000 UTC m=+1065.889081260" Nov 03 19:02:12.169786 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:12.169761 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:12.171441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:12.171417 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:02:12.171562 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:12.171476 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 19:02:12.195923 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:12.195901 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw"] Nov 03 19:02:12.198681 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:02:12.198653 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d614c12_4321_4857_a576_9552dc037c70.slice/crio-6db90f28d18f3d55c7489fb6749a2e1d4a982ccfb2eb12302883b082dc180cbf WatchSource:0}: Error finding container 6db90f28d18f3d55c7489fb6749a2e1d4a982ccfb2eb12302883b082dc180cbf: Status 404 returned error can't find the container with id 6db90f28d18f3d55c7489fb6749a2e1d4a982ccfb2eb12302883b082dc180cbf Nov 03 19:02:12.201737 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:12.201692 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:02:12.201850 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:12.201796 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:02:12.201916 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:12.201846 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:02:12.808307 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:12.808267 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" event={"ID":"4d614c12-4321-4857-a576-9552dc037c70","Type":"ContainerStarted","Data":"05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6"} Nov 03 19:02:12.808511 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:12.808353 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" event={"ID":"4d614c12-4321-4857-a576-9552dc037c70","Type":"ContainerStarted","Data":"6db90f28d18f3d55c7489fb6749a2e1d4a982ccfb2eb12302883b082dc180cbf"} Nov 03 19:02:12.923868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:12.923811 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" podStartSLOduration=1.9237959249999999 podStartE2EDuration="1.923795925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:12.921172076 +0000 UTC m=+1066.893733259" watchObservedRunningTime="2026-11-03 19:02:12.923795925 +0000 UTC m=+1066.896357107" Nov 03 19:02:13.177348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:13.174553 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:02:13.177348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:13.174626 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 19:02:13.780152 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:13.780117 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:13.780521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:13.780500 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:02:13.780690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:13.780667 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 19:02:14.172755 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:14.172720 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:02:14.173375 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:14.173342 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 19:02:14.775783 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:14.775744 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:02:14.776058 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:14.776031 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 19:02:15.184407 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:15.175736 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:02:15.184407 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:15.175791 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 19:02:15.783062 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:15.783023 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:02:15.787526 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:15.787410 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 19:02:16.177408 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:16.170394 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:02:16.177408 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:16.170450 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 19:02:16.783354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:16.778729 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:02:16.783354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:16.778797 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 19:02:17.175351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:17.171706 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:02:17.175351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:17.171769 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 19:02:17.779797 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:17.779752 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:02:17.780085 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:17.780052 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 19:02:18.191498 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:18.191460 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:02:18.192009 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:18.191522 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 19:02:18.775068 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:18.775018 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:02:18.775287 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:18.775106 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 19:02:19.415685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:19.415625 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:19.416478 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:19.416448 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:19.430005 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:19.429981 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:02:20.313607 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:20.313567 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:20.313989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:20.313970 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:20.314805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:20.314780 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:02:26.945932 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:26.945895 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh"] Nov 03 19:02:26.951132 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:26.951108 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.000057 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.000031 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-010ed3-openshift-default-dockercfg-9tvsf\"" Nov 03 19:02:27.037533 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.037505 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lknf7\" (UniqueName: \"kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-kube-api-access-lknf7\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.037693 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.037588 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.037693 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.037638 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.037803 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.037775 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.037867 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.037855 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.037909 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.037874 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.037975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.037955 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.038023 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.038009 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.038068 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.038038 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.049255 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.049225 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh"] Nov 03 19:02:27.139714 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.139680 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.139899 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.139730 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.139899 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.139752 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.139899 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.139787 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lknf7\" (UniqueName: \"kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-kube-api-access-lknf7\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.139899 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.139849 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.139899 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.139889 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.140160 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.139921 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.140160 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.139973 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.140160 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.139997 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.140319 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.140209 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.140319 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.140284 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.140423 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.140402 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.140493 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.140470 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.140669 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.140651 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.142427 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.142401 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.142628 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.142609 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.200055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.199988 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.200055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.200035 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lknf7\" (UniqueName: \"kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-kube-api-access-lknf7\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.265461 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.265428 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:27.688486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.688439 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh"] Nov 03 19:02:27.720799 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:02:27.720765 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887ebeb2_b7c1_45e4_98d2_a125ff72b729.slice/crio-db2a648d9e8b112cc141044585b56b17dad0309e33abbce0fe03d8ba5af40bea WatchSource:0}: Error finding container db2a648d9e8b112cc141044585b56b17dad0309e33abbce0fe03d8ba5af40bea: Status 404 returned error can't find the container with id db2a648d9e8b112cc141044585b56b17dad0309e33abbce0fe03d8ba5af40bea Nov 03 19:02:27.723140 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.723109 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:02:27.723219 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.723177 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:02:27.723219 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.723206 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:02:27.923882 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.923843 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" event={"ID":"887ebeb2-b7c1-45e4-98d2-a125ff72b729","Type":"ContainerStarted","Data":"3f6fc9666989d35f7ca27fa3d0f6ae15282ccbb3a2e2fb634693e20f774c07c0"} Nov 03 19:02:27.924075 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:27.923933 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" event={"ID":"887ebeb2-b7c1-45e4-98d2-a125ff72b729","Type":"ContainerStarted","Data":"db2a648d9e8b112cc141044585b56b17dad0309e33abbce0fe03d8ba5af40bea"} Nov 03 19:02:28.038904 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:28.038788 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" podStartSLOduration=2.038769718 podStartE2EDuration="2.038769718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:28.037190803 +0000 UTC m=+1082.009751986" watchObservedRunningTime="2026-11-03 19:02:28.038769718 +0000 UTC m=+1082.011330901" Nov 03 19:02:28.266380 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:28.266321 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:28.268297 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:28.268272 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:02:28.268453 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:28.268322 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 19:02:29.267727 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:29.267690 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:02:29.268315 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:29.268284 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 19:02:30.269352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:30.268737 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:02:30.269352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:30.268804 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 19:02:31.290352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:31.285094 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:02:31.290352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:31.285160 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 19:02:32.268747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:32.268708 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:02:32.269167 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:32.269139 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 19:02:33.275350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:33.273719 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:02:33.275350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:33.273799 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 19:02:34.266750 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:34.266710 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:02:34.266959 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:34.266807 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 19:02:35.465986 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:35.465811 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:35.466596 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:35.466575 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:35.467839 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:35.467814 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:02:41.297186 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:02:41.297149 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:02:43.929649 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:43.929607 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs"] Nov 03 19:02:43.930197 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:43.929814 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" podUID="5740ed16-bc9e-46d7-8157-5267888355d6" containerName="istio-proxy" containerID="cri-o://a71ecfba7a1fd457dcc08d5f82aec0919103d2433074224eee61bec2dcfdd3df" gracePeriod=30 Nov 03 19:02:49.098771 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.098732 5019 generic.go:356] "Generic (PLEG): container finished" podID="5740ed16-bc9e-46d7-8157-5267888355d6" containerID="a71ecfba7a1fd457dcc08d5f82aec0919103d2433074224eee61bec2dcfdd3df" exitCode=0 Nov 03 19:02:49.099259 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.098803 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" event={"ID":"5740ed16-bc9e-46d7-8157-5267888355d6","Type":"ContainerDied","Data":"a71ecfba7a1fd457dcc08d5f82aec0919103d2433074224eee61bec2dcfdd3df"} Nov 03 19:02:49.224046 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.224018 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 19:02:49.366052 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366018 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-credential-socket\") pod \"5740ed16-bc9e-46d7-8157-5267888355d6\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " Nov 03 19:02:49.366052 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366057 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5740ed16-bc9e-46d7-8157-5267888355d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5740ed16-bc9e-46d7-8157-5267888355d6-istio-podinfo\") pod \"5740ed16-bc9e-46d7-8157-5267888355d6\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " Nov 03 19:02:49.366447 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366089 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-istio-token\" (UniqueName: \"kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-istio-token\") pod \"5740ed16-bc9e-46d7-8157-5267888355d6\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " Nov 03 19:02:49.366447 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366126 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-certs\") pod \"5740ed16-bc9e-46d7-8157-5267888355d6\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " Nov 03 19:02:49.366447 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366231 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-data\") pod \"5740ed16-bc9e-46d7-8157-5267888355d6\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " Nov 03 19:02:49.366447 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366271 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-envoy\") pod \"5740ed16-bc9e-46d7-8157-5267888355d6\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " Nov 03 19:02:49.366447 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366311 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-socket\") pod \"5740ed16-bc9e-46d7-8157-5267888355d6\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " Nov 03 19:02:49.366447 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366339 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-credential-socket" pod "5740ed16-bc9e-46d7-8157-5267888355d6" (UID: "5740ed16-bc9e-46d7-8157-5267888355d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:02:49.366447 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366418 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5740ed16-bc9e-46d7-8157-5267888355d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5740ed16-bc9e-46d7-8157-5267888355d6-istiod-ca-cert\") pod \"5740ed16-bc9e-46d7-8157-5267888355d6\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " Nov 03 19:02:49.366888 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366501 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-kube-api-access-c87nn\" (UniqueName: \"kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-kube-api-access-c87nn\") pod \"5740ed16-bc9e-46d7-8157-5267888355d6\" (UID: \"5740ed16-bc9e-46d7-8157-5267888355d6\") " Nov 03 19:02:49.366888 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366551 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-certs" pod "5740ed16-bc9e-46d7-8157-5267888355d6" (UID: "5740ed16-bc9e-46d7-8157-5267888355d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:02:49.366888 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366869 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5740ed16-bc9e-46d7-8157-5267888355d6-istiod-ca-cert" pod "5740ed16-bc9e-46d7-8157-5267888355d6" (UID: "5740ed16-bc9e-46d7-8157-5267888355d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:02:49.367057 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.367026 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-data" pod "5740ed16-bc9e-46d7-8157-5267888355d6" (UID: "5740ed16-bc9e-46d7-8157-5267888355d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:02:49.367116 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.367038 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:49.367116 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.367084 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:49.367116 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.367101 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5740ed16-bc9e-46d7-8157-5267888355d6-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:49.367266 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.366701 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-socket" pod "5740ed16-bc9e-46d7-8157-5267888355d6" (UID: "5740ed16-bc9e-46d7-8157-5267888355d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:02:49.368755 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.368730 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-envoy" pod "5740ed16-bc9e-46d7-8157-5267888355d6" (UID: "5740ed16-bc9e-46d7-8157-5267888355d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:02:49.368955 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.368933 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5740ed16-bc9e-46d7-8157-5267888355d6-istio-podinfo" pod "5740ed16-bc9e-46d7-8157-5267888355d6" (UID: "5740ed16-bc9e-46d7-8157-5267888355d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:02:49.369310 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.369291 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-kube-api-access-c87nn" pod "5740ed16-bc9e-46d7-8157-5267888355d6" (UID: "5740ed16-bc9e-46d7-8157-5267888355d6"). InnerVolumeSpecName "kube-api-access-c87nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:02:49.369427 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.369400 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-istio-token" pod "5740ed16-bc9e-46d7-8157-5267888355d6" (UID: "5740ed16-bc9e-46d7-8157-5267888355d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:02:49.468075 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.467992 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5740ed16-bc9e-46d7-8157-5267888355d6-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:49.468075 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.468018 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:49.468075 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.468030 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:49.468075 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.468039 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:49.468075 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.468047 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5740ed16-bc9e-46d7-8157-5267888355d6-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:49.468075 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:49.468056 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c87nn\" (UniqueName: \"kubernetes.io/projected/5740ed16-bc9e-46d7-8157-5267888355d6-kube-api-access-c87nn\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:50.108062 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:50.108033 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" Nov 03 19:02:50.108521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:50.108090 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs" event={"ID":"5740ed16-bc9e-46d7-8157-5267888355d6","Type":"ContainerDied","Data":"9c68c0b81e627ebd98bf7eca002e07f6935ec0a8138bfd4ccae364c015d0ce55"} Nov 03 19:02:50.108521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:50.108165 5019 scope.go:122] "RemoveContainer" containerID="a71ecfba7a1fd457dcc08d5f82aec0919103d2433074224eee61bec2dcfdd3df" Nov 03 19:02:50.308958 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:50.308925 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs"] Nov 03 19:02:50.365861 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:50.365732 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z89fs"] Nov 03 19:02:51.356488 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.356454 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp"] Nov 03 19:02:51.357061 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.357031 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5740ed16-bc9e-46d7-8157-5267888355d6" containerName="istio-proxy" Nov 03 19:02:51.357061 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.357046 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5740ed16-bc9e-46d7-8157-5267888355d6" containerName="istio-proxy" Nov 03 19:02:51.357257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.357210 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5740ed16-bc9e-46d7-8157-5267888355d6" containerName="istio-proxy" Nov 03 19:02:51.361977 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.361960 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.429552 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.429523 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-98860e\"/\"kube-root-ca.crt\"" Nov 03 19:02:51.474402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.474373 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-98860e\"/\"openshift-service-ca.crt\"" Nov 03 19:02:51.474641 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.474617 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-98860e\"/\"default-dockercfg-nmqfn\"" Nov 03 19:02:51.474819 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.474571 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-98860e\"/\"distinct-w4-98860e-kserve-self-signed-certs\"" Nov 03 19:02:51.478747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.478724 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp"] Nov 03 19:02:51.500726 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.500690 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-model-cache\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.501040 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.501019 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-home\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.501199 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.501177 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4k7\" (UniqueName: \"kubernetes.io/projected/7cc38985-372c-4466-8e01-1413d5a581d9-kube-api-access-lm4k7\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.501413 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.501386 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-dshm\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.501545 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.501527 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-tmp-dir\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.501743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.501725 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc38985-372c-4466-8e01-1413d5a581d9-tls-certs\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.619223 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.619132 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc38985-372c-4466-8e01-1413d5a581d9-tls-certs\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.620053 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.620029 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-model-cache\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.620734 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.620713 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-home\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.620887 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.620868 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4k7\" (UniqueName: \"kubernetes.io/projected/7cc38985-372c-4466-8e01-1413d5a581d9-kube-api-access-lm4k7\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.621059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.621043 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-dshm\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.621187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.621170 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-tmp-dir\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.621891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.621870 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-home\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.622311 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.620579 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-model-cache\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.622606 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.622579 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-tmp-dir\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.637203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.637175 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc38985-372c-4466-8e01-1413d5a581d9-tls-certs\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.668352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.665661 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-dshm\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.707429 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.696131 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4k7\" (UniqueName: \"kubernetes.io/projected/7cc38985-372c-4466-8e01-1413d5a581d9-kube-api-access-lm4k7\") pod \"distinct-w4-98860e-kserve-65fddf785c-mtxwp\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:51.979349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:51.976554 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:52.201458 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:52.201425 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5740ed16-bc9e-46d7-8157-5267888355d6" path="/var/lib/kubelet/pods/5740ed16-bc9e-46d7-8157-5267888355d6/volumes" Nov 03 19:02:53.106352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:53.098791 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp"] Nov 03 19:02:53.252098 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:53.252024 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" event={"ID":"7cc38985-372c-4466-8e01-1413d5a581d9","Type":"ContainerStarted","Data":"c3cacbe926099768d9908b07b91de1e682062e52aa7b194286d9892ed0646aa2"} Nov 03 19:02:54.260673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:54.260634 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" event={"ID":"7cc38985-372c-4466-8e01-1413d5a581d9","Type":"ContainerStarted","Data":"414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222"} Nov 03 19:02:54.372889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:54.372838 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podStartSLOduration=3.37282497 podStartE2EDuration="3.37282497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:54.37019864 +0000 UTC m=+1108.342759823" watchObservedRunningTime="2026-11-03 19:02:54.37282497 +0000 UTC m=+1108.345386166" Nov 03 19:02:54.977533 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:54.977495 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:02:56.808742 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.808700 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47"] Nov 03 19:02:56.813645 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.813624 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.894275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.894240 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.894493 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.894320 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbcxg\" (UniqueName: \"kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-kube-api-access-lbcxg\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.894568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.894520 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.894621 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.894589 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.894674 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.894618 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.894738 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.894712 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.894793 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.894766 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.894847 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.894803 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.894901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.894853 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.996523 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.996488 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.996723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.996542 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.996723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.996572 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.996723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.996589 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.996723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.996641 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.996723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.996694 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.996991 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.996757 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbcxg\" (UniqueName: \"kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-kube-api-access-lbcxg\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.996991 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.996801 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.997147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.997121 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.997366 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.997194 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.997517 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.997492 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.997617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.997531 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.997617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.997203 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.997726 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.997618 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.999136 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.999115 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:56.999386 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:56.999364 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:57.021889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:57.021862 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47"] Nov 03 19:02:57.054418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:57.054395 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:57.054646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:57.054624 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbcxg\" (UniqueName: \"kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-kube-api-access-lbcxg\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:57.127581 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:57.127497 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:57.552863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:57.552678 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47"] Nov 03 19:02:57.553896 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:02:57.553869 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc320ff_47d9_44dc_b85f_27848a66a7ec.slice/crio-aedbff0eb92efba4293fdbe05765429fc341a1ffd0f5bcacf47827672019bc5c WatchSource:0}: Error finding container aedbff0eb92efba4293fdbe05765429fc341a1ffd0f5bcacf47827672019bc5c: Status 404 returned error can't find the container with id aedbff0eb92efba4293fdbe05765429fc341a1ffd0f5bcacf47827672019bc5c Nov 03 19:02:57.556546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:57.556514 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:02:57.556619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:57.556579 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:02:57.556619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:57.556608 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:02:58.291609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:58.291572 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" event={"ID":"9fc320ff-47d9-44dc-b85f-27848a66a7ec","Type":"ContainerStarted","Data":"15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd"} Nov 03 19:02:58.292010 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:58.291630 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" event={"ID":"9fc320ff-47d9-44dc-b85f-27848a66a7ec","Type":"ContainerStarted","Data":"aedbff0eb92efba4293fdbe05765429fc341a1ffd0f5bcacf47827672019bc5c"} Nov 03 19:02:58.455506 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:58.455451 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" podStartSLOduration=2.45543672 podStartE2EDuration="2.45543672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:58.452149106 +0000 UTC m=+1112.424710284" watchObservedRunningTime="2026-11-03 19:02:58.45543672 +0000 UTC m=+1112.427997903" Nov 03 19:02:59.127925 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:59.127885 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:02:59.129724 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:59.129699 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:02:59.129834 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:02:59.129746 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" podUID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 19:03:00.128157 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:00.128121 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:03:00.128575 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:00.128180 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" podUID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 19:03:01.128406 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:01.128362 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:03:01.128808 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:01.128428 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" podUID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 19:03:01.980578 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:01.980544 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:03:01.996472 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:01.996443 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:03:02.128427 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:02.128394 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:03:02.128810 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:02.128446 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" podUID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 19:03:02.326219 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:02.326133 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:03:03.128817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:03.128785 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:03:03.129207 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:03.128841 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" podUID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 19:03:04.377396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:04.377354 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:03:04.377867 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:04.377844 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:03:04.378622 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:04.378604 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:03:10.969074 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:10.968999 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l"] Nov 03 19:03:10.974191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:10.974173 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.076349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.076288 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l"] Nov 03 19:03:11.091407 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.091366 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.091563 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.091418 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.091563 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.091521 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.091701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.091588 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.091759 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.091701 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.091816 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.091791 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.091889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.091856 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.091951 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.091907 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xljkc\" (UniqueName: \"kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-kube-api-access-xljkc\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.092007 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.091970 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6f7402-2537-47a7-abc6-b21101126c4e-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.196084 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.196043 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.196406 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.196379 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.196612 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.196589 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.196702 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.196624 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.196702 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.196651 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xljkc\" (UniqueName: \"kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-kube-api-access-xljkc\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.196702 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.196689 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6f7402-2537-47a7-abc6-b21101126c4e-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.196849 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.196778 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.196849 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.196807 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.196946 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.196872 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.197776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.197751 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.199198 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.199178 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.199814 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.199491 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.199937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.199756 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6f7402-2537-47a7-abc6-b21101126c4e-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.200117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.200100 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.213092 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.213059 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.220279 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.220198 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.291674 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.289263 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.293149 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.293122 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xljkc\" (UniqueName: \"kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-kube-api-access-xljkc\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:11.406601 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:03:11.406567 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:03:11.601398 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:11.588783 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:12.195363 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:12.180930 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:12.195363 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:12.181021 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:12.195363 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:12.181071 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:12.294652 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:12.293917 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l"] Nov 03 19:03:12.435549 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:12.435505 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" event={"ID":"9d6f7402-2537-47a7-abc6-b21101126c4e","Type":"ContainerStarted","Data":"13a65c293f5d6fff40466b6091e7f99c052e50098fb5385b4ccb9d350da12069"} Nov 03 19:03:12.722400 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:12.706965 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp"] Nov 03 19:03:12.863832 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:12.857386 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp"] Nov 03 19:03:12.863832 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:12.857515 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.010440 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.010030 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.010440 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.010094 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.010440 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.010156 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7kj\" (UniqueName: \"kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-kube-api-access-kr7kj\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.011072 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.011042 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.011202 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.011125 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.011202 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.011163 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.011202 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.011193 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.011412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.011249 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.011412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.011287 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.123350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.122559 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.123350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.122664 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7kj\" (UniqueName: \"kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-kube-api-access-kr7kj\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.123350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.122747 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.123673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.123634 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.130678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.123737 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.130678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.124183 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.130678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.124320 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.130678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.124406 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.130678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.124546 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.130678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.125197 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.130678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.129655 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.130678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.130314 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.130678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.124034 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.130678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.130619 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.139872 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.139820 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.143644 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.143599 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.182711 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.182650 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7kj\" (UniqueName: \"kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-kube-api-access-kr7kj\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.220797 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.219850 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.444685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.444645 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" event={"ID":"9d6f7402-2537-47a7-abc6-b21101126c4e","Type":"ContainerStarted","Data":"c8a8e39be54d67c70e4404ea82b40a05d54c16bdec0fc99751e4ea8ca7c341da"} Nov 03 19:03:13.498763 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.498725 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:13.594349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.589614 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:13.594349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.592434 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:03:13.594349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.592501 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 19:03:13.620845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:13.619175 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" podStartSLOduration=3.619153562 podStartE2EDuration="3.619153562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:13.564816501 +0000 UTC m=+1127.537377684" watchObservedRunningTime="2026-11-03 19:03:13.619153562 +0000 UTC m=+1127.591714745" Nov 03 19:03:14.098833 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.098798 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp"] Nov 03 19:03:14.099198 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:03:14.099172 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c37240b_0a43_4d38_a54e_47fdb0ff5c69.slice/crio-1e2f305007f955d6719ca1ed03bb52dbb0349dd685c4b46af7f11bfe539e129c WatchSource:0}: Error finding container 1e2f305007f955d6719ca1ed03bb52dbb0349dd685c4b46af7f11bfe539e129c: Status 404 returned error can't find the container with id 1e2f305007f955d6719ca1ed03bb52dbb0349dd685c4b46af7f11bfe539e129c Nov 03 19:03:14.101930 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.101895 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:14.102024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.101961 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:14.102024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.102020 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:14.189708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.186712 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w"] Nov 03 19:03:14.192356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.192309 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.246527 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.246495 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvttn\" (UniqueName: \"kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-kube-api-access-nvttn\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.246882 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.246550 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.246882 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.246715 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.246882 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.246812 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c8abc6-7671-4fc0-908a-c56b56f135f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.247043 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.246903 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.247043 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.246993 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.247043 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.247019 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.247136 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.247059 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.247136 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.247119 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.250531 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.250501 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w"] Nov 03 19:03:14.348962 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.348877 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvttn\" (UniqueName: \"kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-kube-api-access-nvttn\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.348962 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.348926 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.348962 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.348962 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.349239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.348996 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c8abc6-7671-4fc0-908a-c56b56f135f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.349239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.349024 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.349239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.349092 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.349239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.349117 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.349239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.349136 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.349239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.349158 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.349553 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.349477 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.349695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.349645 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.349695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.349658 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.350087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.350068 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c8abc6-7671-4fc0-908a-c56b56f135f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.350247 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.350234 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.351235 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.351212 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.352015 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.351991 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.406429 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.406399 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvttn\" (UniqueName: \"kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-kube-api-access-nvttn\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.406584 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.406435 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-qn57w\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.453035 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.452998 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" event={"ID":"5c37240b-0a43-4d38-a54e-47fdb0ff5c69","Type":"ContainerStarted","Data":"dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae"} Nov 03 19:03:14.453183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.453061 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" event={"ID":"5c37240b-0a43-4d38-a54e-47fdb0ff5c69","Type":"ContainerStarted","Data":"1e2f305007f955d6719ca1ed03bb52dbb0349dd685c4b46af7f11bfe539e129c"} Nov 03 19:03:14.500551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.500500 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:14.509239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.509210 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:14.626561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.597793 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:03:14.626561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.597849 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 19:03:14.626561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.609596 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" podStartSLOduration=2.609573033 podStartE2EDuration="2.609573033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:14.565902326 +0000 UTC m=+1128.538463509" watchObservedRunningTime="2026-11-03 19:03:14.609573033 +0000 UTC m=+1128.582134219" Nov 03 19:03:15.009243 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:15.002154 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:15.009243 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:15.002248 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:15.009243 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:15.002291 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:15.009243 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:14.986655 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w"] Nov 03 19:03:15.545371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:15.540784 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:03:15.545371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:15.540857 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" podUID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:03:15.578957 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:15.577790 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" event={"ID":"45c8abc6-7671-4fc0-908a-c56b56f135f3","Type":"ContainerStarted","Data":"0a48ecb2c181d9f95a5b1a13a5e9997ab58893e54c282550c481748745c637cc"} Nov 03 19:03:15.578957 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:15.577884 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" event={"ID":"45c8abc6-7671-4fc0-908a-c56b56f135f3","Type":"ContainerStarted","Data":"94035224675c925fa406f2adcfcab0c84b87125213619893524078546821a747"} Nov 03 19:03:15.590051 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:15.590017 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:03:15.590307 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:15.590278 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 19:03:16.505943 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:16.505911 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:03:16.506270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:16.506226 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" podUID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:03:16.511542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:16.511519 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:16.512233 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:16.512197 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qn57w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:03:16.512421 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:16.512397 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" podUID="45c8abc6-7671-4fc0-908a-c56b56f135f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 19:03:16.648096 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:16.648041 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:03:16.648732 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:16.648680 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 19:03:17.500885 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:17.500849 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:03:17.501194 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:17.501150 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" podUID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:03:17.509784 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:17.509755 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qn57w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:03:17.510048 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:17.510010 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" podUID="45c8abc6-7671-4fc0-908a-c56b56f135f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 19:03:17.603829 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:17.603796 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:03:17.604752 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:17.604720 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 19:03:18.502350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:18.501766 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:03:18.502350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:18.501834 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" podUID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:03:18.513788 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:18.513754 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qn57w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:03:18.514021 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:18.513994 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" podUID="45c8abc6-7671-4fc0-908a-c56b56f135f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 19:03:18.596352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:18.591864 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:03:18.596352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:18.591919 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 19:03:19.512352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:19.502728 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:03:19.512352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:19.502798 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" podUID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:03:19.520360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:19.518658 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qn57w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:03:19.520360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:19.518718 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" podUID="45c8abc6-7671-4fc0-908a-c56b56f135f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 19:03:19.616352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:19.603551 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:03:19.616352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:19.603609 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 19:03:20.507350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:20.506771 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:03:20.507350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:20.506832 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" podUID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:03:20.510361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:20.509775 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qn57w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:03:20.510361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:20.509849 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" podUID="45c8abc6-7671-4fc0-908a-c56b56f135f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 19:03:20.609516 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:20.609480 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:20.610772 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:20.610749 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:20.613533 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:20.613489 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:03:20.727187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:20.727131 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" podStartSLOduration=6.727115117 podStartE2EDuration="6.727115117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:15.703763218 +0000 UTC m=+1129.676324401" watchObservedRunningTime="2026-11-03 19:03:20.727115117 +0000 UTC m=+1134.699676300" Nov 03 19:03:21.505452 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:21.505422 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:21.505896 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:21.505880 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:21.506631 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:21.506610 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:03:21.509885 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:21.509861 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qn57w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:03:21.509978 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:21.509910 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" podUID="45c8abc6-7671-4fc0-908a-c56b56f135f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 19:03:22.510319 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:22.510276 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qn57w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:03:22.510744 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:22.510383 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" podUID="45c8abc6-7671-4fc0-908a-c56b56f135f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 19:03:24.010461 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:24.010417 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:24.011314 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:24.011292 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:24.012570 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:24.012549 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:03:26.712146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.712111 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq"] Nov 03 19:03:26.717837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.717816 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.788132 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.788088 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.788353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.788206 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c83d496d-1fb9-4860-897c-b312023a1b47-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.788353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.788267 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.788497 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.788401 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqhjt\" (UniqueName: \"kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-kube-api-access-fqhjt\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.788497 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.788430 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.788497 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.788462 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c83d496d-1fb9-4860-897c-b312023a1b47-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.788497 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.788493 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.788710 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.788515 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.788710 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.788553 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.816670 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.816633 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq"] Nov 03 19:03:26.889619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.889584 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqhjt\" (UniqueName: \"kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-kube-api-access-fqhjt\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.889619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.889622 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.889863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.889645 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c83d496d-1fb9-4860-897c-b312023a1b47-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.889863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.889674 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.889863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.889691 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.889863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.889722 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.889863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.889784 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.890124 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.889878 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c83d496d-1fb9-4860-897c-b312023a1b47-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.890124 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.889940 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.890364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.890311 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.890451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.890363 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.890659 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.890638 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.890727 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.890705 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.890946 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.890921 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c83d496d-1fb9-4860-897c-b312023a1b47-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.891915 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.891890 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.892267 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.892250 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c83d496d-1fb9-4860-897c-b312023a1b47-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.968937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.968856 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:26.972261 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:26.972240 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqhjt\" (UniqueName: \"kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-kube-api-access-fqhjt\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:27.030836 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.030805 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:27.411716 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.411626 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p"] Nov 03 19:03:27.418050 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.418024 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.455385 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.455362 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq"] Nov 03 19:03:27.457536 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:03:27.457511 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83d496d_1fb9_4860_897c_b312023a1b47.slice/crio-d8ec4ad3e950af94e63c9b0e78ba760dc412eaa2262a16510810ff0245d457bc WatchSource:0}: Error finding container d8ec4ad3e950af94e63c9b0e78ba760dc412eaa2262a16510810ff0245d457bc: Status 404 returned error can't find the container with id d8ec4ad3e950af94e63c9b0e78ba760dc412eaa2262a16510810ff0245d457bc Nov 03 19:03:27.460185 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.460145 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:27.460283 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.460220 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:27.460283 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.460255 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:27.473485 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.473459 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p"] Nov 03 19:03:27.497558 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.497527 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6pj\" (UniqueName: \"kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-kube-api-access-6m6pj\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.497653 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.497597 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.497734 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.497711 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.497780 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.497762 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.497836 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.497822 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.497925 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.497910 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.497962 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.497946 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.497998 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.497972 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.498041 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.498015 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.600567 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.600531 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.600776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.600581 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.600776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.600609 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.600776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.600651 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.600776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.600748 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6pj\" (UniqueName: \"kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-kube-api-access-6m6pj\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.601095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.600802 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.601095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.600857 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.601095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.600888 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.601095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.600944 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.601413 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.601242 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.601561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.601537 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.602122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.601934 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.602122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.601998 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.602840 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.602814 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.604808 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.604777 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.605104 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.605084 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.658134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.658099 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.658300 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.658259 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6pj\" (UniqueName: \"kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-kube-api-access-6m6pj\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.704743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.704711 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" event={"ID":"c83d496d-1fb9-4860-897c-b312023a1b47","Type":"ContainerStarted","Data":"94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4"} Nov 03 19:03:27.704894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.704769 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" event={"ID":"c83d496d-1fb9-4860-897c-b312023a1b47","Type":"ContainerStarted","Data":"d8ec4ad3e950af94e63c9b0e78ba760dc412eaa2262a16510810ff0245d457bc"} Nov 03 19:03:27.732227 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.732196 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:27.820598 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:27.820537 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" podStartSLOduration=1.8205172790000002 podStartE2EDuration="1.820517279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:27.817870698 +0000 UTC m=+1141.790431887" watchObservedRunningTime="2026-11-03 19:03:27.820517279 +0000 UTC m=+1141.793078462" Nov 03 19:03:28.031877 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:28.031797 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:28.033803 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:28.033771 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:03:28.033962 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:28.033832 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 19:03:28.209848 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:28.192670 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p"] Nov 03 19:03:28.247555 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:28.245197 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:28.247555 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:28.245296 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:28.247555 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:28.245357 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:28.713416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:28.713361 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" event={"ID":"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320","Type":"ContainerStarted","Data":"e4f9609cdf4e853bcb33396638e197795c6c706d5fcfe83ba1fcfe708026a298"} Nov 03 19:03:28.713623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:28.713437 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" event={"ID":"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320","Type":"ContainerStarted","Data":"5a6068ba105e7a5b4a81e8365494771a7da813aee9dedda2b1e42528b2f957ca"} Nov 03 19:03:28.733085 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:28.733054 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:28.832403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:28.832322 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" podStartSLOduration=1.832304446 podStartE2EDuration="1.832304446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:28.82857851 +0000 UTC m=+1142.801139692" watchObservedRunningTime="2026-11-03 19:03:28.832304446 +0000 UTC m=+1142.804865630" Nov 03 19:03:29.031831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.031745 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:03:29.031831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.031797 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 19:03:29.316939 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.316860 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f"] Nov 03 19:03:29.320903 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.320885 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.420115 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.420057 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f"] Nov 03 19:03:29.437370 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.437318 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.437370 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.437377 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfcjr\" (UniqueName: \"kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-kube-api-access-lfcjr\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.437625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.437398 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.437625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.437447 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.437625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.437489 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.437625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.437552 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.437625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.437601 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.437625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.437618 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.437939 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.437658 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.539421 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.539387 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.539626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.539496 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.539626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.539553 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.539709 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.539666 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.539765 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.539748 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.539802 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.539789 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.539868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.539852 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.539928 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.539869 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.539977 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.539960 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.540026 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.539988 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfcjr\" (UniqueName: \"kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-kube-api-access-lfcjr\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.540101 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.540059 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.540168 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.540130 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.540279 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.540219 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.540421 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.540402 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.542339 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.541583 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.543452 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.542628 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.619781 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.619697 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfcjr\" (UniqueName: \"kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-kube-api-access-lfcjr\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.621974 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.621916 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gmc4f\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.636037 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.636014 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:29.788350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.787469 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:03:29.788350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:29.787534 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 19:03:30.031786 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:30.031751 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:03:30.031966 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:30.031802 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 19:03:30.226177 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:30.226138 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f"] Nov 03 19:03:30.241064 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:03:30.241037 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdeb4f8e_b96b_4da7_a8a0_be70c56d66f8.slice/crio-aecd6f67e4436c2f5ac13b3ba8adcf3f832c7b29413233e599038f456bfb0fc9 WatchSource:0}: Error finding container aecd6f67e4436c2f5ac13b3ba8adcf3f832c7b29413233e599038f456bfb0fc9: Status 404 returned error can't find the container with id aecd6f67e4436c2f5ac13b3ba8adcf3f832c7b29413233e599038f456bfb0fc9 Nov 03 19:03:30.243597 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:30.243564 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:30.243672 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:30.243627 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:30.243672 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:30.243655 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:03:30.732724 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:30.732684 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:03:30.732913 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:30.732743 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 19:03:30.779644 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:30.779610 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" event={"ID":"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8","Type":"ContainerStarted","Data":"249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded"} Nov 03 19:03:30.780266 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:30.780238 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" event={"ID":"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8","Type":"ContainerStarted","Data":"aecd6f67e4436c2f5ac13b3ba8adcf3f832c7b29413233e599038f456bfb0fc9"} Nov 03 19:03:31.031926 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:31.031833 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:03:31.032505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:31.032461 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 19:03:31.637859 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:31.637825 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:31.640275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:31.640245 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gmc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:03:31.640500 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:31.640475 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" podUID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 19:03:31.733352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:31.732889 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:03:31.733352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:31.732967 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 19:03:32.032287 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:32.031823 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:03:32.032287 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:32.031888 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 19:03:32.645351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:32.639647 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gmc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:03:32.645351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:32.639707 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" podUID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 19:03:32.733758 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:32.733716 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:03:32.734038 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:32.734013 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 19:03:33.037858 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:33.037821 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:03:33.038506 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:33.038456 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 19:03:33.637303 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:33.637263 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gmc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:03:33.637543 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:33.637358 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" podUID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 19:03:33.732917 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:33.732878 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:03:33.733213 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:33.733182 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 19:03:34.036350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:34.035870 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:03:34.036350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:34.035931 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 19:03:34.636406 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:34.636370 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gmc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:03:34.636934 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:34.636901 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" podUID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 19:03:34.733043 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:34.733001 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:03:34.733234 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:34.733073 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 19:03:35.082219 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:35.065683 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:35.134670 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:35.134639 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:35.142644 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:35.142615 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:03:35.203153 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:35.203102 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" podStartSLOduration=6.203088863 podStartE2EDuration="6.203088863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:30.904645242 +0000 UTC m=+1144.877206424" watchObservedRunningTime="2026-11-03 19:03:35.203088863 +0000 UTC m=+1149.175650044" Nov 03 19:03:35.636864 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:35.636832 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gmc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:03:35.637306 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:35.636889 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" podUID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 19:03:35.737418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:35.737380 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:35.737811 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:35.737785 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:35.738455 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:35.738434 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:03:36.636760 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:36.636720 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gmc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:03:36.636962 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:36.636776 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" podUID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 19:03:37.637018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:37.636976 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gmc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:03:37.637426 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:37.637031 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" podUID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 19:03:39.121808 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:39.121768 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:39.122614 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:39.122576 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:39.123387 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:39.123365 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:03:40.764382 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.764346 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-8ks5x-qsr7m"] Nov 03 19:03:40.768892 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.768871 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:40.864698 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.864646 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-l6xrr\"" Nov 03 19:03:40.869744 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.869713 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8ks5x-qsr7m"] Nov 03 19:03:40.878032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.878002 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-trusted-ca-bundle\") pod \"periodic-gathering-8ks5x-qsr7m\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:40.878178 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.878058 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddrkn\" (UniqueName: \"kubernetes.io/projected/f7a2ab97-cf49-4951-adb6-759b7605718a-kube-api-access-ddrkn\") pod \"periodic-gathering-8ks5x-qsr7m\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:40.878239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.878185 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-service-ca-bundle\") pod \"periodic-gathering-8ks5x-qsr7m\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:40.878358 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.878294 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7a2ab97-cf49-4951-adb6-759b7605718a-archives-path\") pod \"periodic-gathering-8ks5x-qsr7m\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:40.979323 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.979291 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7a2ab97-cf49-4951-adb6-759b7605718a-archives-path\") pod \"periodic-gathering-8ks5x-qsr7m\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:40.979481 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.979440 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-trusted-ca-bundle\") pod \"periodic-gathering-8ks5x-qsr7m\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:40.979539 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.979479 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddrkn\" (UniqueName: \"kubernetes.io/projected/f7a2ab97-cf49-4951-adb6-759b7605718a-kube-api-access-ddrkn\") pod \"periodic-gathering-8ks5x-qsr7m\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:40.979539 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.979515 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-service-ca-bundle\") pod \"periodic-gathering-8ks5x-qsr7m\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:40.980059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.979644 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7a2ab97-cf49-4951-adb6-759b7605718a-archives-path\") pod \"periodic-gathering-8ks5x-qsr7m\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:40.980124 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.980115 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-service-ca-bundle\") pod \"periodic-gathering-8ks5x-qsr7m\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:40.980389 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:40.980366 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-trusted-ca-bundle\") pod \"periodic-gathering-8ks5x-qsr7m\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:41.037227 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:41.037156 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddrkn\" (UniqueName: \"kubernetes.io/projected/f7a2ab97-cf49-4951-adb6-759b7605718a-kube-api-access-ddrkn\") pod \"periodic-gathering-8ks5x-qsr7m\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:41.081838 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:41.081812 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:03:41.296789 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:03:41.296704 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:03:41.452709 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:41.452519 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8ks5x-qsr7m"] Nov 03 19:03:41.453057 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:03:41.453034 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a2ab97_cf49_4951_adb6_759b7605718a.slice/crio-b788a30ced3437b8cbe12cdda862b6d29b12f303715e513ce467a106be1ba62b WatchSource:0}: Error finding container b788a30ced3437b8cbe12cdda862b6d29b12f303715e513ce467a106be1ba62b: Status 404 returned error can't find the container with id b788a30ced3437b8cbe12cdda862b6d29b12f303715e513ce467a106be1ba62b Nov 03 19:03:41.984384 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:41.984316 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" event={"ID":"f7a2ab97-cf49-4951-adb6-759b7605718a","Type":"ContainerStarted","Data":"4956bebf40e3a49e9d4ac06da2cd7ea53b48661ff05022fdca96604181cdb565"} Nov 03 19:03:41.984763 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:41.984406 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" event={"ID":"f7a2ab97-cf49-4951-adb6-759b7605718a","Type":"ContainerStarted","Data":"b788a30ced3437b8cbe12cdda862b6d29b12f303715e513ce467a106be1ba62b"} Nov 03 19:03:42.094396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:03:42.094341 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" podStartSLOduration=2.094312738 podStartE2EDuration="2.094312738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:42.093446151 +0000 UTC m=+1156.066007314" watchObservedRunningTime="2026-11-03 19:03:42.094312738 +0000 UTC m=+1156.066873920" Nov 03 19:04:11.318134 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:04:11.318096 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:04:21.824801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:21.824412 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:04:21.824801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:21.824489 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:04:26.270311 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:26.270276 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:04:26.272866 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:26.272846 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:04:41.299203 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:04:41.299121 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:04:42.305760 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.305726 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q"] Nov 03 19:04:42.311024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.311002 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.392010 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.391966 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q"] Nov 03 19:04:42.454593 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.454555 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a160d419-2de3-4dc1-affb-e50a0867becb-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.454593 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.454594 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.454873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.454724 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.454873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.454764 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.454873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.454786 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.455036 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.454874 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a160d419-2de3-4dc1-affb-e50a0867becb-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.455036 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.454990 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.455036 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.455026 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.455193 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.455148 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrtxb\" (UniqueName: \"kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-kube-api-access-qrtxb\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.556246 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.556160 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a160d419-2de3-4dc1-affb-e50a0867becb-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.556246 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.556195 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.556466 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.556267 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.556466 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.556301 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.556562 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.556474 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.556562 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.556548 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a160d419-2de3-4dc1-affb-e50a0867becb-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.556669 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.556641 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.556734 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.556670 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.556790 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.556741 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.556790 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.556775 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrtxb\" (UniqueName: \"kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-kube-api-access-qrtxb\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.556901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.556841 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.556958 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.556884 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.557063 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.557043 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.557400 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.557364 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a160d419-2de3-4dc1-affb-e50a0867becb-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.558937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.558915 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a160d419-2de3-4dc1-affb-e50a0867becb-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.559044 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.559000 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.614342 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.614298 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.614514 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.614487 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrtxb\" (UniqueName: \"kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-kube-api-access-qrtxb\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:42.624781 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:42.624759 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:43.054258 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:43.054225 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q"] Nov 03 19:04:43.054505 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:04:43.054475 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda160d419_2de3_4dc1_affb_e50a0867becb.slice/crio-3bc9819ba2bd7d4e082e00b7f79f34f4666124158e75eb471f093f4514170361 WatchSource:0}: Error finding container 3bc9819ba2bd7d4e082e00b7f79f34f4666124158e75eb471f093f4514170361: Status 404 returned error can't find the container with id 3bc9819ba2bd7d4e082e00b7f79f34f4666124158e75eb471f093f4514170361 Nov 03 19:04:43.057472 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:43.057428 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:04:43.057575 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:43.057518 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:04:43.057575 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:43.057564 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:04:43.559623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:43.559584 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" event={"ID":"a160d419-2de3-4dc1-affb-e50a0867becb","Type":"ContainerStarted","Data":"76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437"} Nov 03 19:04:43.559998 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:43.559648 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" event={"ID":"a160d419-2de3-4dc1-affb-e50a0867becb","Type":"ContainerStarted","Data":"3bc9819ba2bd7d4e082e00b7f79f34f4666124158e75eb471f093f4514170361"} Nov 03 19:04:43.624858 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:43.624828 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:43.674306 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:43.674256 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" podStartSLOduration=1.674242894 podStartE2EDuration="1.674242894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:43.671555508 +0000 UTC m=+1217.644116689" watchObservedRunningTime="2026-11-03 19:04:43.674242894 +0000 UTC m=+1217.646804076" Nov 03 19:04:44.626344 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:44.626301 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 19:04:44.626770 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:44.626396 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" podUID="a160d419-2de3-4dc1-affb-e50a0867becb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 19:04:45.625100 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:45.625063 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 19:04:45.625269 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:45.625134 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" podUID="a160d419-2de3-4dc1-affb-e50a0867becb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 19:04:46.625855 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:46.625317 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 19:04:46.625855 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:46.625420 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" podUID="a160d419-2de3-4dc1-affb-e50a0867becb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 19:04:47.722111 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:47.722071 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 19:04:47.722800 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:47.722764 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" podUID="a160d419-2de3-4dc1-affb-e50a0867becb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 19:04:48.629352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:48.626954 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 19:04:48.629352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:48.627031 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" podUID="a160d419-2de3-4dc1-affb-e50a0867becb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 19:04:49.625280 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:49.625195 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 19:04:49.625280 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:49.625256 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" podUID="a160d419-2de3-4dc1-affb-e50a0867becb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 19:04:50.629120 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:50.629081 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:50.629614 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:50.629599 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:50.630300 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:50.630281 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:04:53.713961 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.713926 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj"] Nov 03 19:04:53.718639 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.718615 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.763124 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.763080 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-home\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.763300 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.763257 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tmp-dir\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.763404 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.763310 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmfx\" (UniqueName: \"kubernetes.io/projected/1d172ec5-a832-4cba-ae9f-10b54f54b10b-kube-api-access-7pmfx\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.763466 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.763405 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-model-cache\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.763523 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.763475 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-dshm\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.763614 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.763587 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tls-certs\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.814305 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.814273 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-98860e\"/\"distinct-2-w4-98860e-kserve-self-signed-certs\"" Nov 03 19:04:53.819573 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.819538 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj"] Nov 03 19:04:53.864528 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.864499 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-home\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.864681 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.864584 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tmp-dir\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.864681 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.864608 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmfx\" (UniqueName: \"kubernetes.io/projected/1d172ec5-a832-4cba-ae9f-10b54f54b10b-kube-api-access-7pmfx\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.864681 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.864637 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-model-cache\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.864681 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.864673 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-dshm\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.864894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.864712 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tls-certs\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.864968 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.864937 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-home\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.865019 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.864986 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-model-cache\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.865019 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.865005 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tmp-dir\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.867038 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.867013 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-dshm\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.867191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.867173 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tls-certs\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:53.947851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:53.947815 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmfx\" (UniqueName: \"kubernetes.io/projected/1d172ec5-a832-4cba-ae9f-10b54f54b10b-kube-api-access-7pmfx\") pod \"distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:54.032827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:54.032741 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:04:54.451107 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:54.451048 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj"] Nov 03 19:04:54.468868 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:04:54.468838 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d172ec5_a832_4cba_ae9f_10b54f54b10b.slice/crio-124f517931090fc208356279527aa32b5b59444f9222231b678f695650b862b7 WatchSource:0}: Error finding container 124f517931090fc208356279527aa32b5b59444f9222231b678f695650b862b7: Status 404 returned error can't find the container with id 124f517931090fc208356279527aa32b5b59444f9222231b678f695650b862b7 Nov 03 19:04:54.908470 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:54.908379 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" event={"ID":"1d172ec5-a832-4cba-ae9f-10b54f54b10b","Type":"ContainerStarted","Data":"390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e"} Nov 03 19:04:54.908470 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:54.908431 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" event={"ID":"1d172ec5-a832-4cba-ae9f-10b54f54b10b","Type":"ContainerStarted","Data":"124f517931090fc208356279527aa32b5b59444f9222231b678f695650b862b7"} Nov 03 19:04:55.020161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:55.020110 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podStartSLOduration=2.020097028 podStartE2EDuration="2.020097028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:55.0176147 +0000 UTC m=+1228.990175885" watchObservedRunningTime="2026-11-03 19:04:55.020097028 +0000 UTC m=+1228.992658210" Nov 03 19:04:55.043667 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:04:55.043637 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:05:01.479159 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.479110 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk"] Nov 03 19:05:01.484234 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.484206 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.570220 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.570184 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"openshift-service-ca.crt\"" Nov 03 19:05:01.570438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.570281 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"kube-root-ca.crt\"" Nov 03 19:05:01.579960 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.579919 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"default-dockercfg-fbmhw\"" Nov 03 19:05:01.580116 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.580002 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"distinct-w5-90a9bf-kserve-self-signed-certs\"" Nov 03 19:05:01.583485 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.583456 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk"] Nov 03 19:05:01.681220 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.681179 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfh5l\" (UniqueName: \"kubernetes.io/projected/33b49433-77e0-4513-bc54-c8d3d10dc443-kube-api-access-mfh5l\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.682136 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.682104 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-model-cache\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.691500 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.691463 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b49433-77e0-4513-bc54-c8d3d10dc443-tls-certs\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.693244 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.693215 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-dshm\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.693579 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.693549 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-home\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.693747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.693731 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-tmp-dir\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.795308 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.795218 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-dshm\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.795308 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.795278 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-home\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.795574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.795319 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-tmp-dir\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.795574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.795460 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfh5l\" (UniqueName: \"kubernetes.io/projected/33b49433-77e0-4513-bc54-c8d3d10dc443-kube-api-access-mfh5l\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.795685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.795608 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-model-cache\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.795685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.795657 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b49433-77e0-4513-bc54-c8d3d10dc443-tls-certs\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.795837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.795796 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-tmp-dir\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.795954 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.795934 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-home\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.796119 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.795980 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-model-cache\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.797620 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.797596 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-dshm\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.798117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.798095 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b49433-77e0-4513-bc54-c8d3d10dc443-tls-certs\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:01.873412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:01.854244 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfh5l\" (UniqueName: \"kubernetes.io/projected/33b49433-77e0-4513-bc54-c8d3d10dc443-kube-api-access-mfh5l\") pod \"distinct-w5-90a9bf-kserve-74ffbb655-sdrjk\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:02.098481 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:02.098385 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:02.474577 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:02.474547 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk"] Nov 03 19:05:02.475021 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:05:02.474989 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33b49433_77e0_4513_bc54_c8d3d10dc443.slice/crio-c44e9aac873796f1572d2ada67aa6c27e8334691b5cc857ec4455652e4762110 WatchSource:0}: Error finding container c44e9aac873796f1572d2ada67aa6c27e8334691b5cc857ec4455652e4762110: Status 404 returned error can't find the container with id c44e9aac873796f1572d2ada67aa6c27e8334691b5cc857ec4455652e4762110 Nov 03 19:05:03.005946 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:03.005905 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" event={"ID":"33b49433-77e0-4513-bc54-c8d3d10dc443","Type":"ContainerStarted","Data":"cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d"} Nov 03 19:05:03.006358 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:03.005954 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" event={"ID":"33b49433-77e0-4513-bc54-c8d3d10dc443","Type":"ContainerStarted","Data":"c44e9aac873796f1572d2ada67aa6c27e8334691b5cc857ec4455652e4762110"} Nov 03 19:05:03.099133 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:03.099100 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:03.119281 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:03.119229 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podStartSLOduration=2.119208744 podStartE2EDuration="2.119208744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:03.116178269 +0000 UTC m=+1237.088739452" watchObservedRunningTime="2026-11-03 19:05:03.119208744 +0000 UTC m=+1237.091769928" Nov 03 19:05:04.047351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:04.046999 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:05:04.089354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:04.081466 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:05:05.052122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:05.052091 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:05:11.296900 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:05:11.296865 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:05:12.098831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.098798 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:12.111757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.111732 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:12.834481 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.834449 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76"] Nov 03 19:05:12.838582 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.838563 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:12.918157 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.918122 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:12.918368 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.918204 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:12.918368 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.918307 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fe1063-da3d-4283-a161-a9ce010d56fd-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:12.918542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.918401 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5g5p\" (UniqueName: \"kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-kube-api-access-c5g5p\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:12.918542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.918495 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:12.918542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.918534 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:12.918656 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.918581 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:12.918689 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.918656 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:12.918689 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:12.918672 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.019867 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.019830 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020027 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.019890 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020027 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.019919 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fe1063-da3d-4283-a161-a9ce010d56fd-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020027 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.019949 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5g5p\" (UniqueName: \"kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-kube-api-access-c5g5p\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.020097 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020171 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.020156 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020232 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.020214 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020287 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.020237 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.020382 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020475 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.020416 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020562 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.020539 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020715 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.020699 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fe1063-da3d-4283-a161-a9ce010d56fd-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020782 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.020721 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.020782 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.020738 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.022253 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.022224 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.022638 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.022620 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.039302 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.039280 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76"] Nov 03 19:05:13.077364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.077321 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.077495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.077475 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5g5p\" (UniqueName: \"kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-kube-api-access-c5g5p\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.103730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.103669 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:05:13.152917 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.152886 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:13.577646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.577618 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76"] Nov 03 19:05:13.578439 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:05:13.578414 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fe1063_da3d_4283_a161_a9ce010d56fd.slice/crio-ed705e35160ea980135615f29a2c0013e9c82f890c336c5705672b053025f5dc WatchSource:0}: Error finding container ed705e35160ea980135615f29a2c0013e9c82f890c336c5705672b053025f5dc: Status 404 returned error can't find the container with id ed705e35160ea980135615f29a2c0013e9c82f890c336c5705672b053025f5dc Nov 03 19:05:13.581027 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.580993 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:05:13.581095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.581058 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:05:13.581095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:13.581089 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:05:14.106519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:14.106473 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" event={"ID":"b1fe1063-da3d-4283-a161-a9ce010d56fd","Type":"ContainerStarted","Data":"e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b"} Nov 03 19:05:14.106914 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:14.106543 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" event={"ID":"b1fe1063-da3d-4283-a161-a9ce010d56fd","Type":"ContainerStarted","Data":"ed705e35160ea980135615f29a2c0013e9c82f890c336c5705672b053025f5dc"} Nov 03 19:05:14.156121 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:14.156089 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:14.157690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:14.157655 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 19:05:14.157837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:14.157725 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 19:05:14.307033 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:14.306979 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" podStartSLOduration=2.3069663670000002 podStartE2EDuration="2.306966367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:14.276845399 +0000 UTC m=+1248.249406584" watchObservedRunningTime="2026-11-03 19:05:14.306966367 +0000 UTC m=+1248.279527548" Nov 03 19:05:15.153769 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:15.153733 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 19:05:15.154177 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:15.153797 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 19:05:16.153941 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:16.153911 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 19:05:16.154380 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:16.153959 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 19:05:17.154133 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:17.154103 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 19:05:17.154593 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:17.154164 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 19:05:18.138433 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:18.138395 5019 generic.go:356] "Generic (PLEG): container finished" podID="f7a2ab97-cf49-4951-adb6-759b7605718a" containerID="4956bebf40e3a49e9d4ac06da2cd7ea53b48661ff05022fdca96604181cdb565" exitCode=0 Nov 03 19:05:18.138595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:18.138462 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" event={"ID":"f7a2ab97-cf49-4951-adb6-759b7605718a","Type":"ContainerDied","Data":"4956bebf40e3a49e9d4ac06da2cd7ea53b48661ff05022fdca96604181cdb565"} Nov 03 19:05:18.153596 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:18.153569 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 19:05:18.153744 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:18.153638 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 19:05:19.153851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.153822 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 19:05:19.154233 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.153877 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 19:05:19.407746 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.407686 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:05:19.534158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.534082 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-trusted-ca-bundle\") pod \"f7a2ab97-cf49-4951-adb6-759b7605718a\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " Nov 03 19:05:19.534341 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.534185 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-service-ca-bundle\") pod \"f7a2ab97-cf49-4951-adb6-759b7605718a\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " Nov 03 19:05:19.534341 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.534256 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a2ab97-cf49-4951-adb6-759b7605718a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7a2ab97-cf49-4951-adb6-759b7605718a-archives-path\") pod \"f7a2ab97-cf49-4951-adb6-759b7605718a\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " Nov 03 19:05:19.534556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.534536 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a2ab97-cf49-4951-adb6-759b7605718a-kube-api-access-ddrkn\" (UniqueName: \"kubernetes.io/projected/f7a2ab97-cf49-4951-adb6-759b7605718a-kube-api-access-ddrkn\") pod \"f7a2ab97-cf49-4951-adb6-759b7605718a\" (UID: \"f7a2ab97-cf49-4951-adb6-759b7605718a\") " Nov 03 19:05:19.534681 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.534651 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-service-ca-bundle" pod "f7a2ab97-cf49-4951-adb6-759b7605718a" (UID: "f7a2ab97-cf49-4951-adb6-759b7605718a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:05:19.534769 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.534744 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-trusted-ca-bundle" pod "f7a2ab97-cf49-4951-adb6-759b7605718a" (UID: "f7a2ab97-cf49-4951-adb6-759b7605718a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:05:19.535044 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.535023 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a2ab97-cf49-4951-adb6-759b7605718a-archives-path" pod "f7a2ab97-cf49-4951-adb6-759b7605718a" (UID: "f7a2ab97-cf49-4951-adb6-759b7605718a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:05:19.535292 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.535272 5019 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-trusted-ca-bundle\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:19.535411 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.535297 5019 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a2ab97-cf49-4951-adb6-759b7605718a-service-ca-bundle\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:19.535411 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.535313 5019 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7a2ab97-cf49-4951-adb6-759b7605718a-archives-path\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:19.536938 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.536911 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a2ab97-cf49-4951-adb6-759b7605718a-kube-api-access-ddrkn" pod "f7a2ab97-cf49-4951-adb6-759b7605718a" (UID: "f7a2ab97-cf49-4951-adb6-759b7605718a"). InnerVolumeSpecName "kube-api-access-ddrkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:05:19.636527 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:19.636489 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddrkn\" (UniqueName: \"kubernetes.io/projected/f7a2ab97-cf49-4951-adb6-759b7605718a-kube-api-access-ddrkn\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:20.152790 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:20.152758 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" Nov 03 19:05:20.152950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:20.152754 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8ks5x-qsr7m" event={"ID":"f7a2ab97-cf49-4951-adb6-759b7605718a","Type":"ContainerDied","Data":"b788a30ced3437b8cbe12cdda862b6d29b12f303715e513ce467a106be1ba62b"} Nov 03 19:05:20.152950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:20.152903 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b788a30ced3437b8cbe12cdda862b6d29b12f303715e513ce467a106be1ba62b" Nov 03 19:05:20.153217 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:20.153194 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 19:05:20.153285 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:20.153260 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 19:05:21.158585 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:21.158548 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:21.159001 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:21.158938 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:21.159808 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:21.159787 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:05:26.046969 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.046144 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc"] Nov 03 19:05:26.046969 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.046918 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f7a2ab97-cf49-4951-adb6-759b7605718a" containerName="insights-gathering" Nov 03 19:05:26.046969 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.046940 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a2ab97-cf49-4951-adb6-759b7605718a" containerName="insights-gathering" Nov 03 19:05:26.047613 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.047159 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a2ab97-cf49-4951-adb6-759b7605718a" containerName="insights-gathering" Nov 03 19:05:26.052442 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.052412 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.135486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.135439 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-tmp-dir\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.135671 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.135600 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-model-cache\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.135758 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.135714 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bk7z\" (UniqueName: \"kubernetes.io/projected/90d072fe-3d22-4a06-9227-87069cf3e96c-kube-api-access-5bk7z\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.135826 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.135781 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d072fe-3d22-4a06-9227-87069cf3e96c-tls-certs\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.135917 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.135875 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-home\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.135997 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.135978 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-dshm\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.151374 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.151279 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fa2878\"/\"default-dockercfg-s6hcs\"" Nov 03 19:05:26.151666 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.151618 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fa2878\"/\"openshift-service-ca.crt\"" Nov 03 19:05:26.152011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.151816 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fa2878\"/\"distinct-w3-fa2878-kserve-self-signed-certs\"" Nov 03 19:05:26.152011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.151895 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fa2878\"/\"kube-root-ca.crt\"" Nov 03 19:05:26.164809 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.164776 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc"] Nov 03 19:05:26.294359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.293854 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-home\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.294359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.294047 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-dshm\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.294359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.294164 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-tmp-dir\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.294359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.294300 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-model-cache\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.294781 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.294401 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bk7z\" (UniqueName: \"kubernetes.io/projected/90d072fe-3d22-4a06-9227-87069cf3e96c-kube-api-access-5bk7z\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.294781 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.294497 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d072fe-3d22-4a06-9227-87069cf3e96c-tls-certs\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.309851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.309757 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-model-cache\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.310567 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.310541 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-tmp-dir\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.311103 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.311081 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-home\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.349892 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.349858 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d072fe-3d22-4a06-9227-87069cf3e96c-tls-certs\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.357744 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.357676 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-dshm\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.387097 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.387035 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bk7z\" (UniqueName: \"kubernetes.io/projected/90d072fe-3d22-4a06-9227-87069cf3e96c-kube-api-access-5bk7z\") pod \"distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:26.670674 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:26.670630 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:27.351918 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.351884 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj"] Nov 03 19:05:27.397552 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.397520 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj"] Nov 03 19:05:27.412104 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.397825 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.420862 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.420811 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj"] Nov 03 19:05:27.421067 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.420927 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.465663 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.465620 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj"] Nov 03 19:05:27.551015 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.550774 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc"] Nov 03 19:05:27.579992 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.579960 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.580116 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580000 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.580116 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580105 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.580237 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580129 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.580237 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580177 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.580237 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580200 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.580430 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580338 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btt5w\" (UniqueName: \"kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-kube-api-access-btt5w\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.580430 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580381 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.580547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580449 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.580547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580529 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.580663 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580565 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.580663 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580582 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.580776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580665 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.580776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580697 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.580776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580729 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.580776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580748 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.580998 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580842 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8qtf\" (UniqueName: \"kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-kube-api-access-h8qtf\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.580998 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.580881 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.682322 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682286 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.682520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682435 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.682520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682488 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.682645 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682536 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8qtf\" (UniqueName: \"kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-kube-api-access-h8qtf\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.682645 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682590 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.682645 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682633 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.682798 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682666 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.682798 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682742 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.682798 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682766 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.682949 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682832 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.682949 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682864 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.682949 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682897 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btt5w\" (UniqueName: \"kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-kube-api-access-btt5w\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.682949 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682898 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.682949 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682926 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.683183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.682980 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.683183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.683055 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.683183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.683084 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.683183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.683099 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.683183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.683104 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.683183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.683174 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.683509 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.683221 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.683604 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.683576 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.683684 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.683603 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.683857 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.683811 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.683961 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.683859 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.684073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.684054 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.684151 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.684057 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.684494 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.684471 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.685155 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.685137 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.685948 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.685919 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.686182 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.686162 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.688674 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.688653 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.740428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.740396 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.740567 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.740488 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.740567 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.740542 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8qtf\" (UniqueName: \"kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-kube-api-access-h8qtf\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:27.740636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.740575 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btt5w\" (UniqueName: \"kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-kube-api-access-btt5w\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:27.755281 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:27.755255 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:28.040217 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.040186 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:28.184036 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:05:28.184011 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748bcc6f_9eed_4a21_acdc_3c9cbdbdf4dd.slice/crio-22c073fed15922fccbc65dfa2e3b1c5e6e5390072ec908ff678a008b4f63c321 WatchSource:0}: Error finding container 22c073fed15922fccbc65dfa2e3b1c5e6e5390072ec908ff678a008b4f63c321: Status 404 returned error can't find the container with id 22c073fed15922fccbc65dfa2e3b1c5e6e5390072ec908ff678a008b4f63c321 Nov 03 19:05:28.184178 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.184109 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj"] Nov 03 19:05:28.187486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.187445 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:05:28.187588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.187530 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:05:28.187588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.187571 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:05:28.397583 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.397475 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" event={"ID":"90d072fe-3d22-4a06-9227-87069cf3e96c","Type":"ContainerStarted","Data":"11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8"} Nov 03 19:05:28.397583 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.397539 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" event={"ID":"90d072fe-3d22-4a06-9227-87069cf3e96c","Type":"ContainerStarted","Data":"71274d0b25989b6f12596076b2c13a2e3ab55b87708e82904e68b4f029e088ea"} Nov 03 19:05:28.399862 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.399834 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" event={"ID":"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd","Type":"ContainerStarted","Data":"3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21"} Nov 03 19:05:28.400010 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.399885 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" event={"ID":"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd","Type":"ContainerStarted","Data":"22c073fed15922fccbc65dfa2e3b1c5e6e5390072ec908ff678a008b4f63c321"} Nov 03 19:05:28.417673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.417628 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj"] Nov 03 19:05:28.418813 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:05:28.418782 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5791d78_831c_4fd7_9d30_1aaa69049ae0.slice/crio-9698c8555b86274da6a5b67c2ee40d6420518039e32c685aba42d9eb990aa6e5 WatchSource:0}: Error finding container 9698c8555b86274da6a5b67c2ee40d6420518039e32c685aba42d9eb990aa6e5: Status 404 returned error can't find the container with id 9698c8555b86274da6a5b67c2ee40d6420518039e32c685aba42d9eb990aa6e5 Nov 03 19:05:28.421959 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.421922 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:05:28.422059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.421992 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:05:28.422059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.422025 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:05:28.510183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.510135 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podStartSLOduration=2.51012106 podStartE2EDuration="2.51012106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:28.506943925 +0000 UTC m=+1262.479505108" watchObservedRunningTime="2026-11-03 19:05:28.51012106 +0000 UTC m=+1262.482682241" Nov 03 19:05:28.672746 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.672434 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:28.760351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:28.756735 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:29.418511 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:29.418473 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" event={"ID":"c5791d78-831c-4fd7-9d30-1aaa69049ae0","Type":"ContainerStarted","Data":"e4a44decea15ea87466514bcba77bb5fd29896f0c599872b0a8de44e321e5d73"} Nov 03 19:05:29.418931 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:29.418558 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" event={"ID":"c5791d78-831c-4fd7-9d30-1aaa69049ae0","Type":"ContainerStarted","Data":"9698c8555b86274da6a5b67c2ee40d6420518039e32c685aba42d9eb990aa6e5"} Nov 03 19:05:29.637280 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:29.637217 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" podStartSLOduration=2.637197742 podStartE2EDuration="2.637197742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:28.621598476 +0000 UTC m=+1262.594159670" watchObservedRunningTime="2026-11-03 19:05:29.637197742 +0000 UTC m=+1263.609758926" Nov 03 19:05:29.638740 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:29.638700 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" podStartSLOduration=2.638685157 podStartE2EDuration="2.638685157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:29.588957209 +0000 UTC m=+1263.561518392" watchObservedRunningTime="2026-11-03 19:05:29.638685157 +0000 UTC m=+1263.611246331" Nov 03 19:05:29.757745 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:29.757661 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:05:29.757745 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:29.757719 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:05:30.040973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:30.040837 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:30.042633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:30.042601 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:05:30.042780 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:30.042655 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 19:05:30.756055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:30.756023 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:05:30.756451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:30.756075 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:05:31.041478 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:31.041388 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:05:31.041478 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:31.041451 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 19:05:31.759757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:31.759717 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:05:31.760391 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:31.760354 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:05:32.041319 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:32.041220 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:05:32.041319 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:32.041277 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 19:05:32.755702 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:32.755663 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:05:32.755907 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:32.755727 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:05:33.042026 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:33.041940 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:05:33.042026 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:33.041990 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 19:05:33.756513 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:33.756466 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:05:33.756740 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:33.756529 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:05:34.040703 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:34.040620 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:05:34.040897 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:34.040878 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 19:05:34.756607 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:34.756564 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:05:34.757029 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:34.756638 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:05:35.040796 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:35.040698 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:05:35.040796 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:35.040764 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 19:05:35.884679 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:35.884643 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:35.886193 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:35.886173 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:35.891412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:35.891392 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:05:36.157019 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:36.156946 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:36.157259 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:36.157241 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:36.158053 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:36.158030 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:05:36.767119 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:36.767085 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:36.779841 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:36.779812 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:37.490207 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:05:37.490176 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:05:41.466316 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:05:41.466275 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:06:11.317523 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:06:11.317440 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:06:27.114710 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:27.114663 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:06:27.131950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:27.131903 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:06:27.686738 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:27.686694 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:06:29.433570 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.433538 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb"] Nov 03 19:06:29.438271 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.438249 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.540399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.540365 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb"] Nov 03 19:06:29.570987 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.570951 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.571172 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.571002 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.571172 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.571102 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.571281 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.571210 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.571281 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.571271 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.571436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.571359 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxdm\" (UniqueName: \"kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-kube-api-access-qbxdm\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.571436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.571418 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.571547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.571490 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.571547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.571534 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.672687 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.672654 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.672883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.672711 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.672883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.672750 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxdm\" (UniqueName: \"kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-kube-api-access-qbxdm\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.672883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.672802 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.672883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.672875 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.673101 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.672915 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.673101 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.673040 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.673101 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.673075 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.673262 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.673104 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.673262 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.673148 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.673634 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.673587 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.673719 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.673627 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.673892 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.673866 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.673980 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.673950 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.675506 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.675485 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.675773 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.675753 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.730089 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.730029 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.730208 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.730168 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxdm\" (UniqueName: \"kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-kube-api-access-qbxdm\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:29.752275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:29.752254 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:30.174535 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:06:30.174501 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59721d1a_3083_4cd0_9f11_dd6c6c862e2c.slice/crio-cb9e0a12cc81b1591d4ec4c062ced8a4ec268b98f16e53f84cc0061c3a295537 WatchSource:0}: Error finding container cb9e0a12cc81b1591d4ec4c062ced8a4ec268b98f16e53f84cc0061c3a295537: Status 404 returned error can't find the container with id cb9e0a12cc81b1591d4ec4c062ced8a4ec268b98f16e53f84cc0061c3a295537 Nov 03 19:06:30.174771 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:30.174748 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb"] Nov 03 19:06:30.177649 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:30.177614 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:06:30.177740 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:30.177715 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:06:30.177810 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:30.177784 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:06:31.166357 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:31.165849 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" event={"ID":"59721d1a-3083-4cd0-9f11-dd6c6c862e2c","Type":"ContainerStarted","Data":"ec81dcb3c4493590940f9f11fc39a2247ea25bc41cd46a8665cf2a4787344bea"} Nov 03 19:06:31.166357 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:31.165929 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" event={"ID":"59721d1a-3083-4cd0-9f11-dd6c6c862e2c","Type":"ContainerStarted","Data":"cb9e0a12cc81b1591d4ec4c062ced8a4ec268b98f16e53f84cc0061c3a295537"} Nov 03 19:06:31.329887 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:31.329839 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" podStartSLOduration=2.329824721 podStartE2EDuration="2.329824721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:31.328567812 +0000 UTC m=+1325.301128995" watchObservedRunningTime="2026-11-03 19:06:31.329824721 +0000 UTC m=+1325.302385902" Nov 03 19:06:31.752567 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:31.752533 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:31.754089 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:31.754060 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 19:06:31.754218 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:31.754112 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 19:06:32.753246 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:32.753207 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 19:06:32.753661 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:32.753272 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 19:06:33.752615 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:33.752577 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 19:06:33.752804 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:33.752637 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 19:06:34.752853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:34.752820 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 19:06:34.753260 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:34.752887 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 19:06:35.752976 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:35.752941 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 19:06:35.753398 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:35.753001 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 19:06:36.753290 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:36.753244 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 19:06:36.753826 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:36.753300 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 19:06:37.817063 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:37.817029 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:37.817582 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:37.817563 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:37.819089 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:37.819067 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:06:41.297428 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:06:41.297388 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:06:43.865482 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.865450 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw"] Nov 03 19:06:43.870054 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.870031 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:43.912133 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.912105 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv"] Nov 03 19:06:43.916251 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.916232 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:43.971903 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.971872 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:43.972051 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.971932 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:43.972051 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.971988 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzfbx\" (UniqueName: \"kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-kube-api-access-jzfbx\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:43.972171 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972086 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:43.972171 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972108 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:43.972171 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972125 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:43.972360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972192 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:43.972360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972210 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:43.972360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972301 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:43.972524 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972369 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:43.972524 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972390 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:43.972524 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972408 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:43.972524 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972494 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:43.972730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972553 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:43.972730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972586 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm6kb\" (UniqueName: \"kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-kube-api-access-hm6kb\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:43.972730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972662 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:43.972896 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972761 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:43.972896 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:43.972804 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.024087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.024055 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv"] Nov 03 19:06:44.077651 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.077195 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw"] Nov 03 19:06:44.078434 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.078402 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.078695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.078527 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.079028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.079001 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.079627 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.079603 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.079754 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.079669 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.079830 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.079780 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.080451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080303 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.080451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080383 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.080451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080414 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzfbx\" (UniqueName: \"kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-kube-api-access-jzfbx\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.080650 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080463 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.080650 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080494 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.080650 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080520 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.080650 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080592 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.080650 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080622 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.080895 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080717 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.080895 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080750 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.080895 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080777 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.080895 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080807 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.080895 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080842 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.081069 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080899 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.081069 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.080933 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm6kb\" (UniqueName: \"kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-kube-api-access-hm6kb\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.083383 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.081536 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.083383 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.082011 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.083383 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.083063 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.083639 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.083437 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.083805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.083746 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.084022 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.084003 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.084617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.084584 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.086574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.086542 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.087037 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.087018 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.092581 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.092558 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.093001 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.092973 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.136508 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.136438 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.137973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.137949 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm6kb\" (UniqueName: \"kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-kube-api-access-hm6kb\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.138431 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.138409 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzfbx\" (UniqueName: \"kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-kube-api-access-jzfbx\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.139478 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.139452 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.184577 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.184543 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:44.230124 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.230086 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:44.610479 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.610448 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw"] Nov 03 19:06:44.610906 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:06:44.610881 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc06a73_8b9e_48d9_b4b3_266d74b48f98.slice/crio-4d00f014f1b7830188201432b9d747a01184fc90ed6c4996022abee612ebf9f0 WatchSource:0}: Error finding container 4d00f014f1b7830188201432b9d747a01184fc90ed6c4996022abee612ebf9f0: Status 404 returned error can't find the container with id 4d00f014f1b7830188201432b9d747a01184fc90ed6c4996022abee612ebf9f0 Nov 03 19:06:44.613513 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.613475 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:06:44.613616 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.613545 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:06:44.613616 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.613582 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:06:44.658576 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.658549 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv"] Nov 03 19:06:44.660969 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.660939 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:06:44.661050 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.661001 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:06:44.661050 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:44.661030 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:06:45.417402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:45.417307 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" event={"ID":"4c7e0ab3-5865-4566-a231-ad0b0c05810d","Type":"ContainerStarted","Data":"4e5858b0912ec3d7144bab6b112b6475f8f37dbe2415226757caf8f7974cedf3"} Nov 03 19:06:45.417839 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:45.417420 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" event={"ID":"4c7e0ab3-5865-4566-a231-ad0b0c05810d","Type":"ContainerStarted","Data":"0a21e82e84962b40c776f6e7df80bb22be3f92bd37baeadff22d09015d18b27b"} Nov 03 19:06:45.419578 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:45.419549 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" event={"ID":"dfc06a73-8b9e-48d9-b4b3-266d74b48f98","Type":"ContainerStarted","Data":"8f17092bd2660c8a2809dae05a62fcc51b760ee12368632d8b89d8670f562368"} Nov 03 19:06:45.419683 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:45.419598 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" event={"ID":"dfc06a73-8b9e-48d9-b4b3-266d74b48f98","Type":"ContainerStarted","Data":"4d00f014f1b7830188201432b9d747a01184fc90ed6c4996022abee612ebf9f0"} Nov 03 19:06:45.641828 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:45.641779 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" podStartSLOduration=2.641764566 podStartE2EDuration="2.641764566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:45.593685324 +0000 UTC m=+1339.566246508" watchObservedRunningTime="2026-11-03 19:06:45.641764566 +0000 UTC m=+1339.614325747" Nov 03 19:06:45.715095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:45.714975 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" podStartSLOduration=2.714952565 podStartE2EDuration="2.714952565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:45.70761086 +0000 UTC m=+1339.680172043" watchObservedRunningTime="2026-11-03 19:06:45.714952565 +0000 UTC m=+1339.687513774" Nov 03 19:06:46.196356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:46.194422 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:46.217956 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:46.217478 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:06:46.217956 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:46.217541 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:06:46.230785 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:46.230752 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:46.232027 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:46.231999 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 19:06:46.232149 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:46.232063 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 19:06:47.189348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:47.188777 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:06:47.189348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:47.188848 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:06:47.230789 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:47.230760 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 19:06:47.231006 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:47.230986 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 19:06:48.198349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:48.197811 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:06:48.198349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:48.197873 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:06:48.231516 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:48.231473 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 19:06:48.231669 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:48.231539 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 19:06:49.186349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:49.185730 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:06:49.186349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:49.185795 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:06:49.231668 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:49.231629 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 19:06:49.232152 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:49.232117 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 19:06:50.186351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:50.186201 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:06:50.186351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:50.186277 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:06:50.230802 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:50.230772 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 19:06:50.231011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:50.230990 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 19:06:51.185530 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:51.185490 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:06:51.186042 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:51.185555 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:06:51.231007 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:51.230970 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 19:06:51.231198 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:51.231024 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 19:06:52.481143 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:52.480998 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:52.484400 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:52.482779 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:52.492355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:52.490625 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:06:52.539349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:52.538685 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:52.539749 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:52.539728 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:06:52.542592 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:06:52.542567 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:07:05.047469 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:05.046422 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:05.048084 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:05.048051 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:05.099850 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:05.099807 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:05.325556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:05.325452 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:05.461770 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:05.461456 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:07:05.461770 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:05.461517 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:05.488841 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:05.488458 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:05.552649 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:05.552600 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:06.049973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.047420 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:06.049973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.048298 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:07:06.100800 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.100728 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:06.101798 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.101776 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:07:06.530348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.524016 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:06.563494 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.563449 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:06.563786 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.563764 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:07:06.563945 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.563929 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Nov 03 19:07:06.596630 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.595813 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:07:06.596630 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.595888 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:07:06.596630 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.595989 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:07:06.639393 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.619430 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:07:06.639393 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.619514 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:06.867797 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.827573 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:07:06.893684 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.875492 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:07:06.946098 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.946068 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:07:06.949398 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.947738 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:07:06.997753 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:06.997723 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:07:11.298434 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:07:11.298396 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:07:28.847669 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:28.847636 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d"] Nov 03 19:07:28.852383 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:28.852361 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:28.926128 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:28.926088 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:28.926303 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:28.926250 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:28.926303 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:28.926299 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:28.926444 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:28.926428 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:28.926492 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:28.926478 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:28.926565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:28.926552 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:28.926626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:28.926614 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tvw6\" (UniqueName: \"kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-kube-api-access-9tvw6\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:28.926680 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:28.926643 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:28.926737 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:28.926723 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:28.955073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:28.955026 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d"] Nov 03 19:07:29.027953 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.027918 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.028146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.027967 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.028146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.028011 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.028146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.028046 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.028146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.028102 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.028441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.028173 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvw6\" (UniqueName: \"kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-kube-api-access-9tvw6\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.028441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.028213 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.028441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.028253 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.028441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.028373 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.028644 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.028452 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.028838 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.028806 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.028964 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.028825 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.028964 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.028867 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.029044 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.028971 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.030485 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.030464 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.031255 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.031225 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.085709 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.085675 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.085873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.085847 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvw6\" (UniqueName: \"kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-kube-api-access-9tvw6\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.169303 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.169272 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:29.590497 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.590413 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d"] Nov 03 19:07:29.623007 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:07:29.622981 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode039bad7_2408_4ef8_8d43_4e3a540a6c56.slice/crio-7257f0f987959b85fc5ea1a9f7ccc43c4f27519db51475b3bde7e6fb902f6aa5 WatchSource:0}: Error finding container 7257f0f987959b85fc5ea1a9f7ccc43c4f27519db51475b3bde7e6fb902f6aa5: Status 404 returned error can't find the container with id 7257f0f987959b85fc5ea1a9f7ccc43c4f27519db51475b3bde7e6fb902f6aa5 Nov 03 19:07:29.625802 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.625765 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:07:29.625905 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.625850 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:07:29.625905 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.625895 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:07:29.899540 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.899454 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8"] Nov 03 19:07:29.904251 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.904229 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:29.940084 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.940054 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:29.940224 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.940097 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:29.940224 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.940117 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:29.940314 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.940263 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:29.940314 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.940293 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:29.940440 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.940383 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:29.940527 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.940501 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tprr\" (UniqueName: \"kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-kube-api-access-2tprr\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:29.940643 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.940626 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb92e12-7aec-430a-9063-fbfe91123c44-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:29.940693 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:29.940678 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.013385 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.012194 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8"] Nov 03 19:07:30.043385 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.043347 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.043685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.043666 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.043891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.043865 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.044011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.043919 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.044011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.043944 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.044132 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.044010 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.044132 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.044043 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.044132 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.044104 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.044299 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.044181 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tprr\" (UniqueName: \"kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-kube-api-access-2tprr\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.044299 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.044276 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb92e12-7aec-430a-9063-fbfe91123c44-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.044299 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.044283 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.044703 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.044680 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.045792 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.045763 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb92e12-7aec-430a-9063-fbfe91123c44-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.046070 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.046052 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.048069 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.048046 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.048191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.048176 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.102422 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.102391 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.102633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.102610 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tprr\" (UniqueName: \"kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-kube-api-access-2tprr\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.133106 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.133074 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" event={"ID":"e039bad7-2408-4ef8-8d43-4e3a540a6c56","Type":"ContainerStarted","Data":"b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d"} Nov 03 19:07:30.133255 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.133124 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" event={"ID":"e039bad7-2408-4ef8-8d43-4e3a540a6c56","Type":"ContainerStarted","Data":"7257f0f987959b85fc5ea1a9f7ccc43c4f27519db51475b3bde7e6fb902f6aa5"} Nov 03 19:07:30.169949 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.169911 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:30.171471 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.171447 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:30.171598 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.171507 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:30.222992 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.221724 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:30.797845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.797748 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podStartSLOduration=2.7977254670000002 podStartE2EDuration="2.797725467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:30.265571247 +0000 UTC m=+1384.238132431" watchObservedRunningTime="2026-11-03 19:07:30.797725467 +0000 UTC m=+1384.770286652" Nov 03 19:07:30.804749 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.804715 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8"] Nov 03 19:07:30.816609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.816571 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:07:30.816710 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.816638 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:07:30.816710 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:30.816669 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:07:31.142410 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:31.142288 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" event={"ID":"fbb92e12-7aec-430a-9063-fbfe91123c44","Type":"ContainerStarted","Data":"d47b9bc25fc1102d91b3f4a49771bba0867b041c48db062fe8b2b48a641ba383"} Nov 03 19:07:31.142816 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:31.142442 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" event={"ID":"fbb92e12-7aec-430a-9063-fbfe91123c44","Type":"ContainerStarted","Data":"b024c4c1242a0acd590a84d8c1d05d3522cec85fd363e62e5de626d464bef92c"} Nov 03 19:07:31.169792 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:31.169754 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:31.170099 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:31.170070 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:31.222860 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:31.222823 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:31.224970 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:31.224940 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:31.225101 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:31.224985 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:31.281351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:31.274860 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podStartSLOduration=2.274838634 podStartE2EDuration="2.274838634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:31.259603366 +0000 UTC m=+1385.232164550" watchObservedRunningTime="2026-11-03 19:07:31.274838634 +0000 UTC m=+1385.247399817" Nov 03 19:07:32.189352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:32.182851 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:32.189352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:32.182922 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:32.222917 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:32.222792 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:32.222917 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:32.222858 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:33.179352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:33.171726 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:33.179352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:33.171791 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:33.227889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:33.227843 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:33.228480 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:33.228452 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:34.180895 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:34.180856 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:34.181168 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:34.181137 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:34.222731 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:34.222700 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:34.222953 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:34.222929 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:35.172440 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:35.172403 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:35.173034 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:35.173003 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:35.228016 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:35.227984 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:35.228246 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:35.228221 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:35.888582 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:35.888535 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:36.006466 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:36.006423 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:36.180363 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:36.177886 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:36.180363 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:36.177942 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:36.242167 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:36.241611 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:36.242167 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:36.241665 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:36.352493 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:36.352428 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:36.904351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:36.820416 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:36.914472 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:36.908788 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:36.926361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:36.918492 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:07:36.965381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:36.953189 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:36.965381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:36.953750 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:37.009201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.007011 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:37.009201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.008306 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:07:37.205547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.205496 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:37.206175 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.206129 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:37.259947 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.259905 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:37.260225 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.260189 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:37.826520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.826469 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded" Nov 03 19:07:37.922815 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.830899 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:37.923924 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.919020 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:37.925263 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.925243 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:07:37.960705 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.960658 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:37.986767 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.986716 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:37.988391 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.988367 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:07:38.012965 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:38.012832 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:07:38.013454 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:37.988781 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:07:38.203807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:38.203686 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:38.203807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:38.203741 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:38.244894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:38.244429 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:38.244894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:38.244489 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:39.042348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:39.036729 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:07:39.063413 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:39.062789 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:39.097097 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:39.097041 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:07:39.373410 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:39.341634 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 19:07:39.373410 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:39.342848 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:07:39.407352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:39.401708 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:39.407352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:39.401771 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:39.566364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:39.497551 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:39.566364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:39.497618 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:40.059358 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:40.056611 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:40.155355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:40.144186 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:07:40.311461 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:40.299564 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:40.500586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:40.500541 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:40.515451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:40.365317 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:07:40.538403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:40.409506 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:40.573043 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:40.363439 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:40.621662 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:40.339551 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:40.732222 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:40.732177 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:41.073509 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:41.073453 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:41.150528 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:41.150474 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:41.173928 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:41.173889 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:41.185466 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:41.185421 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:41.223846 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:41.223724 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:41.223846 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:41.223786 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:41.407408 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:41.397472 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:07:41.407408 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:41.397541 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:41.551376 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:41.525099 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:41.620386 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:41.575495 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 19:07:41.638587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:41.638076 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:07:41.918562 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:07:41.918523 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:07:42.215042 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:42.198804 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:07:42.216568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:42.216525 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:07:42.235352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:42.223406 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:07:42.235594 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:42.100319 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:42.327154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:42.327118 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:42.362168 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:42.362116 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:42.552431 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:42.526405 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:42.585724 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:42.585667 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:42.659361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:42.646412 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:42.905717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:42.905631 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:07:43.143681 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:43.140773 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:43.144917 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:43.144890 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:07:43.217423 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:43.217304 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:43.217926 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:43.217893 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:43.288122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:43.288087 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:43.288386 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:43.288357 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:43.436270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:43.435387 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:07:43.534782 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:43.534701 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:07:43.548212 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:43.547932 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:07:43.978290 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:43.978251 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:43.979201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:43.979175 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:43.979942 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:43.979922 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:07:44.222612 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:44.222574 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:07:44.222809 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:44.222652 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:07:45.222839 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:45.222796 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:46.264525 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:46.264488 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:07:46.265001 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:46.264969 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:46.307018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:46.306974 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:46.309366 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:46.309319 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:07:58.747566 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.747530 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl"] Nov 03 19:07:58.752227 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.752207 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.792988 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.792955 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.793156 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.793039 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.793156 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.793130 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.793252 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.793175 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcef2b9-e070-4b5b-9902-323feda55bdf-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.793252 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.793214 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zklh4\" (UniqueName: \"kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-kube-api-access-zklh4\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.793346 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.793293 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.793407 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.793373 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.793521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.793496 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.793799 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.793779 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.853166 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.853127 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl"] Nov 03 19:07:58.895070 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.895032 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.895400 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.895166 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.895400 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.895298 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.895400 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.895367 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.895608 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.895431 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.895608 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.895459 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.895608 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.895486 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcef2b9-e070-4b5b-9902-323feda55bdf-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.895608 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.895517 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zklh4\" (UniqueName: \"kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-kube-api-access-zklh4\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.895608 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.895575 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.895861 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.895629 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.896254 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.896224 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.896652 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.896610 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.896931 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.896912 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.897070 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.897046 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcef2b9-e070-4b5b-9902-323feda55bdf-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.898665 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.898614 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.899550 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.899530 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.953608 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.953577 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:58.953775 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:58.953640 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zklh4\" (UniqueName: \"kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-kube-api-access-zklh4\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:59.066563 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:59.066485 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:07:59.494014 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:59.493985 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl"] Nov 03 19:07:59.494383 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:07:59.494320 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcef2b9_e070_4b5b_9902_323feda55bdf.slice/crio-359933c19d47f24899fd395d19923abc2575c022395bb5f44992c6fa4df54405 WatchSource:0}: Error finding container 359933c19d47f24899fd395d19923abc2575c022395bb5f44992c6fa4df54405: Status 404 returned error can't find the container with id 359933c19d47f24899fd395d19923abc2575c022395bb5f44992c6fa4df54405 Nov 03 19:07:59.497251 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:59.497208 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:07:59.497349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:59.497280 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:07:59.497349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:07:59.497312 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:08:00.151840 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:00.151809 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" event={"ID":"edcef2b9-e070-4b5b-9902-323feda55bdf","Type":"ContainerStarted","Data":"2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638"} Nov 03 19:08:00.152229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:00.151857 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" event={"ID":"edcef2b9-e070-4b5b-9902-323feda55bdf","Type":"ContainerStarted","Data":"359933c19d47f24899fd395d19923abc2575c022395bb5f44992c6fa4df54405"} Nov 03 19:08:00.280038 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:00.279965 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" podStartSLOduration=2.279945944 podStartE2EDuration="2.279945944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:00.255948799 +0000 UTC m=+1414.228509983" watchObservedRunningTime="2026-11-03 19:08:00.279945944 +0000 UTC m=+1414.252507127" Nov 03 19:08:01.100781 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:01.100746 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:08:01.106480 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:01.106443 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:08:01.106657 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:01.106520 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" podUID="edcef2b9-e070-4b5b-9902-323feda55bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:08:02.067709 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:02.067672 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:08:02.068296 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:02.068263 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" podUID="edcef2b9-e070-4b5b-9902-323feda55bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:08:03.070352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:03.067779 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:08:03.070352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:03.067856 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" podUID="edcef2b9-e070-4b5b-9902-323feda55bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:08:04.073712 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:04.073676 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:08:04.074358 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:04.074307 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" podUID="edcef2b9-e070-4b5b-9902-323feda55bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:08:05.069351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:05.067642 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:08:05.069351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:05.067705 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" podUID="edcef2b9-e070-4b5b-9902-323feda55bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:08:06.067751 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:06.067707 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:08:06.068269 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:06.067780 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" podUID="edcef2b9-e070-4b5b-9902-323feda55bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:08:07.239613 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:07.239565 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:08:07.241179 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:07.241156 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:08:07.241903 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:07.241886 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:08:10.488964 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.488925 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk"] Nov 03 19:08:10.605802 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.605764 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.751805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.713143 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-eb6ef8\"/\"distinct-2-w6-eb6ef8-kserve-self-signed-certs\"" Nov 03 19:08:10.781121 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.753050 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tls-certs\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.808813 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.808693 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2kdp\" (UniqueName: \"kubernetes.io/projected/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-kube-api-access-p2kdp\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.809256 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.809216 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tmp-dir\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.809829 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.809805 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-home\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.811053 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.811011 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-dshm\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.816498 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.816470 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-model-cache\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.935507 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.935464 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-model-cache\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.935792 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.935774 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tls-certs\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.943352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.936051 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2kdp\" (UniqueName: \"kubernetes.io/projected/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-kube-api-access-p2kdp\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.943645 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.943617 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tmp-dir\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.943921 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.943895 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-home\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.944073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.944053 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-dshm\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.956353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.948020 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-model-cache\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.983351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.957669 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tmp-dir\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:10.983351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.957994 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk"] Nov 03 19:08:10.983351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:10.958279 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-home\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:11.017380 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:11.017193 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:11.019908 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:11.019860 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:11.029353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:11.022002 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tls-certs\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:11.120835 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:11.115787 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-dshm\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:11.156362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:11.152477 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2kdp\" (UniqueName: \"kubernetes.io/projected/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-kube-api-access-p2kdp\") pod \"distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:11.168932 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:11.168891 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:11.190361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:11.178673 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:11.219352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:11.191897 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:11.346409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:11.346346 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:08:11.924274 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:08:11.924230 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:08:12.019054 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:12.019008 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:12.039566 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:12.039535 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:08:12.072398 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:12.052829 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:12.072724 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:12.072703 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:08:12.195365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:12.195241 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:12.211347 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:12.197936 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:12.257352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:12.257272 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:08:12.285084 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:12.285054 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:08:12.357470 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:12.357417 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:12.358442 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:12.358420 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:08:13.023400 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:13.016524 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 19:08:13.044080 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:13.043772 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 19:08:13.089711 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:13.089668 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:08:13.146450 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:13.146405 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded" Nov 03 19:08:13.210353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:13.206938 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:08:13.235846 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:13.235788 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:13.445543 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:13.407743 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:14.005227 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:14.005194 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:08:14.104543 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:14.045496 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 19:08:14.104543 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:14.090891 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:14.303970 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:14.303928 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:08:14.474461 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:14.473810 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:14.489942 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:14.489873 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" event={"ID":"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace","Type":"ContainerStarted","Data":"171d174b239256fdb0ed3e6504b53c582d42181ae328f2ab2fbc35256bead0cb"} Nov 03 19:08:14.521825 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:14.521767 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk"] Nov 03 19:08:14.538023 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:14.537997 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:08:14.542735 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:14.542710 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:08:14.544310 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:14.544283 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:08:14.551298 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:14.549166 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:08:15.512142 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:15.512098 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" event={"ID":"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace","Type":"ContainerStarted","Data":"84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16"} Nov 03 19:08:15.623754 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:15.623700 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podStartSLOduration=5.623685364 podStartE2EDuration="5.623685364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:15.620962828 +0000 UTC m=+1429.593524011" watchObservedRunningTime="2026-11-03 19:08:15.623685364 +0000 UTC m=+1429.596246546" Nov 03 19:08:16.198177 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:16.198147 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:21.199102 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:21.199067 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:21.211746 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:21.211718 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:21.570437 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:21.570350 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:28.342424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.342392 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw"] Nov 03 19:08:28.347204 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.347180 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.389089 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.389057 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn"] Nov 03 19:08:28.393497 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.393475 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.425707 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.425676 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85086af0-9f26-4517-88e8-8a3bd2219d2d-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.425855 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.425728 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.425855 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.425754 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbhsx\" (UniqueName: \"kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-kube-api-access-fbhsx\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.425931 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.425850 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.425982 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.425971 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.426028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.426007 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.426082 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.426067 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.426117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.426099 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.426154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.426145 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.450445 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.450413 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw"] Nov 03 19:08:28.503952 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.503917 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn"] Nov 03 19:08:28.527517 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.527476 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.527703 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.527603 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.527703 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.527656 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97pw\" (UniqueName: \"kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-kube-api-access-m97pw\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.527831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.527787 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85086af0-9f26-4517-88e8-8a3bd2219d2d-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.527831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.527823 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.527980 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.527873 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.527980 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.527900 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhsx\" (UniqueName: \"kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-kube-api-access-fbhsx\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.528088 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.527993 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.528088 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528031 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48828f6f-de20-4e61-ad91-c648281f8304-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.528243 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528086 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.528243 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528114 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.528243 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528195 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.529568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528250 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.529568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528578 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85086af0-9f26-4517-88e8-8a3bd2219d2d-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.529568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528670 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.529568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528680 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.529568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528738 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.529568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528746 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.529568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528807 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.529568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528856 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48828f6f-de20-4e61-ad91-c648281f8304-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.529568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528886 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.529568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.528917 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.529568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.529115 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.531322 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.531297 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.534568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.534544 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.586773 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.586740 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbhsx\" (UniqueName: \"kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-kube-api-access-fbhsx\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.591195 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.591154 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.634201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.634062 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48828f6f-de20-4e61-ad91-c648281f8304-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.634586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.634481 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.634586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.634543 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.634996 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.634952 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.635312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.635241 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48828f6f-de20-4e61-ad91-c648281f8304-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.635555 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.635505 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.635879 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.635742 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.636406 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.636102 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.636406 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.636383 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48828f6f-de20-4e61-ad91-c648281f8304-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.636406 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.636384 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.636690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.636653 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.636933 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.636908 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97pw\" (UniqueName: \"kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-kube-api-access-m97pw\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.637398 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.637322 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.638277 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.638239 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.640921 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.640898 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.644631 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.644606 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48828f6f-de20-4e61-ad91-c648281f8304-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.661601 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.661576 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:28.692242 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.692215 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.694588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.694562 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97pw\" (UniqueName: \"kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-kube-api-access-m97pw\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:28.709634 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:28.709602 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:29.085484 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.085449 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw"] Nov 03 19:08:29.118588 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:08:29.118559 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85086af0_9f26_4517_88e8_8a3bd2219d2d.slice/crio-4fcb94f732eb91acc7c1188c8232839e35bda3a9406b9fe05d520ea8d5c3980d WatchSource:0}: Error finding container 4fcb94f732eb91acc7c1188c8232839e35bda3a9406b9fe05d520ea8d5c3980d: Status 404 returned error can't find the container with id 4fcb94f732eb91acc7c1188c8232839e35bda3a9406b9fe05d520ea8d5c3980d Nov 03 19:08:29.121471 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.121438 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:08:29.121553 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.121510 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:08:29.121553 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.121540 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:08:29.134615 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:08:29.134587 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48828f6f_de20_4e61_ad91_c648281f8304.slice/crio-98201635882f8cb20be94936818cac753b259c3d46e600df1aba83c233a040e1 WatchSource:0}: Error finding container 98201635882f8cb20be94936818cac753b259c3d46e600df1aba83c233a040e1: Status 404 returned error can't find the container with id 98201635882f8cb20be94936818cac753b259c3d46e600df1aba83c233a040e1 Nov 03 19:08:29.135356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.135201 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn"] Nov 03 19:08:29.138116 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.138082 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:08:29.138208 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.138174 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:08:29.138275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.138209 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:08:29.631502 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.631465 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" event={"ID":"48828f6f-de20-4e61-ad91-c648281f8304","Type":"ContainerStarted","Data":"b3f6d332491b29761a25e91e6b55c4e6cf62e9038914caaccca9cf1e5bc192f3"} Nov 03 19:08:29.632011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.631529 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" event={"ID":"48828f6f-de20-4e61-ad91-c648281f8304","Type":"ContainerStarted","Data":"98201635882f8cb20be94936818cac753b259c3d46e600df1aba83c233a040e1"} Nov 03 19:08:29.633749 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.633714 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" event={"ID":"85086af0-9f26-4517-88e8-8a3bd2219d2d","Type":"ContainerStarted","Data":"71590d9015078414b6fcffe0cce4392a3a3bc59a30f38cf60294d1903c54182f"} Nov 03 19:08:29.633873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.633766 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" event={"ID":"85086af0-9f26-4517-88e8-8a3bd2219d2d","Type":"ContainerStarted","Data":"4fcb94f732eb91acc7c1188c8232839e35bda3a9406b9fe05d520ea8d5c3980d"} Nov 03 19:08:29.661801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.661775 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:29.709811 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.709776 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:29.751006 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.750280 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podStartSLOduration=1.750260573 podStartE2EDuration="1.750260573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:29.745801012 +0000 UTC m=+1443.718362197" watchObservedRunningTime="2026-11-03 19:08:29.750260573 +0000 UTC m=+1443.722821759" Nov 03 19:08:29.868261 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:29.868196 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podStartSLOduration=1.8681801569999998 podStartE2EDuration="1.868180157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:29.867244357 +0000 UTC m=+1443.839805540" watchObservedRunningTime="2026-11-03 19:08:29.868180157 +0000 UTC m=+1443.840741343" Nov 03 19:08:30.224350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.221237 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp"] Nov 03 19:08:30.237359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.229162 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.250378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.245316 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp"] Nov 03 19:08:30.436924 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.436871 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.437129 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.436947 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.437129 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.436991 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.437267 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.437124 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h49zf\" (UniqueName: \"kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-kube-api-access-h49zf\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.437267 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.437207 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.437451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.437304 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.437451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.437436 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.437562 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.437467 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.437562 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.437501 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.538855 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.538767 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.538855 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.538839 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.538855 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.538858 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.539077 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.538879 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.539077 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.538908 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.539077 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.538941 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.539077 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.538972 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.539077 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.539069 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h49zf\" (UniqueName: \"kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-kube-api-access-h49zf\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.539232 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.539131 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.539744 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.539720 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.540277 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.540250 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.540428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.540407 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.540546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.540520 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.540791 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.540767 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.542266 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.542242 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.542504 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.542488 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.636352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.632401 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h49zf\" (UniqueName: \"kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-kube-api-access-h49zf\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.639354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.636826 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:30.683354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.679459 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:30.683354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.679540 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:30.729126 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.728565 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:30.729126 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.728632 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:30.873717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:30.873619 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:31.439795 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:31.439459 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp"] Nov 03 19:08:31.444766 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:08:31.444738 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2389b6ad_dc9d_46e4_9380_3f5595fe53be.slice/crio-fe3840178bc6e01bf774b768ba9bef30ab69285ef487fc8b7d1424b71fd2d714 WatchSource:0}: Error finding container fe3840178bc6e01bf774b768ba9bef30ab69285ef487fc8b7d1424b71fd2d714: Status 404 returned error can't find the container with id fe3840178bc6e01bf774b768ba9bef30ab69285ef487fc8b7d1424b71fd2d714 Nov 03 19:08:31.447278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:31.447241 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:08:31.447416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:31.447387 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:08:31.447486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:31.447464 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:08:31.670385 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:31.665742 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:31.670385 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:31.665806 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:31.843578 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:31.843541 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:31.843737 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:31.843619 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:31.854654 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:31.854625 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" event={"ID":"2389b6ad-dc9d-46e4-9380-3f5595fe53be","Type":"ContainerStarted","Data":"fe3840178bc6e01bf774b768ba9bef30ab69285ef487fc8b7d1424b71fd2d714"} Nov 03 19:08:32.673574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:32.666416 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:32.673574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:32.666483 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:32.729967 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:32.729834 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:32.729967 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:32.729906 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:32.918348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:32.918083 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" event={"ID":"2389b6ad-dc9d-46e4-9380-3f5595fe53be","Type":"ContainerStarted","Data":"1545347abaded93fa19f298a96aff2c457b2c2eaaa7397b07f70d4abf58d296c"} Nov 03 19:08:33.105405 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:33.104962 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podStartSLOduration=3.104941455 podStartE2EDuration="3.104941455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:33.102458936 +0000 UTC m=+1447.075020118" watchObservedRunningTime="2026-11-03 19:08:33.104941455 +0000 UTC m=+1447.077502639" Nov 03 19:08:33.666352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:33.665968 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:33.666352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:33.666049 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:33.713855 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:33.713817 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:33.714474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:33.714442 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:33.891901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:33.891862 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:33.914380 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:33.912180 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:33.914380 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:33.912345 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:34.015353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:34.014581 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:08:34.118345 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:34.115424 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:34.408551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:34.408446 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:34.567351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:34.562517 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:34.567351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:34.563841 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:34.667414 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:34.664745 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:34.667414 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:34.664816 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:34.716421 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:34.715585 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:34.716421 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:34.715663 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:34.898352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:34.876085 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:34.898352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:34.876169 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:35.041410 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.016433 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.041410 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.016858 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:08:35.138350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.135820 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:08:35.138350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.136883 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:08:35.168031 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.167980 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.414526 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.414428 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.444415 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.418767 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:08:35.529353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.493146 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:08:35.566860 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.566835 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:08:35.627315 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.566726 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.634658 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.566680 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.641683 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.636450 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:35.649464 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.649437 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:08:35.986424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.985694 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:35.986424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.985757 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:35.986424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.985839 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:35.986424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.985863 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:35.986424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.985921 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:35.986424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:35.985939 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:36.036351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.030900 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:08:36.241473 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.241365 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:08:36.450360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.437625 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:36.508454 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.462062 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:08:36.657527 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.647879 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 19:08:36.702916 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.688927 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 03 19:08:37.010428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.896909 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:37.010428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.896978 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:37.010428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.901619 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:37.010428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.901679 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:37.010428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.949059 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:08:37.010428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.984656 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:37.010428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:36.984716 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:37.064894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:37.038712 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:37.350455 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:37.328661 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:08:37.463005 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:37.462957 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:08:38.383039 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:38.295730 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:08:38.383039 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:38.171685 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 03 19:08:38.383039 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:38.171747 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded" Nov 03 19:08:38.383039 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:37.478967 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:38.448488 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:38.448436 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 19:08:38.618353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:38.597566 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:38.618353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:38.597649 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:38.618353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:38.597806 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:38.618353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:38.597867 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:38.640428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:38.377513 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:08:38.795576 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:38.795525 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:38.795894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:38.795847 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:39.197556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.102619 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:39.197556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.197156 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:39.197556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.102952 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:39.197556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.197216 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:39.197556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.106545 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:08:39.197556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.197367 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.209483 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.209426 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.297355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.296432 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.303642 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.303428 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.395630 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.392862 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:08:39.395630 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.392925 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.395630 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.393615 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:39.400385 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.393664 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:39.405567 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.405514 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:08:39.527407 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.494413 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.603408 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:08:39.540846 5019 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.378s" Nov 03 19:08:39.603408 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.540958 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:08:39.700721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.667416 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lzknn" podUID="64142b5e-3faa-403f-a7b7-60638cfeb7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 19:08:39.700721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.668048 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" podUID="fc9e70c5-7751-4377-aacf-3c8c8530bdc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Nov 03 19:08:39.748679 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.732919 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:39.748679 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.732980 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:39.865401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.776764 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:39.891875 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.891826 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:39.932391 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:39.932347 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:40.014138 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.014103 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:40.075421 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.075371 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:40.293438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.287424 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:08:40.312828 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.312789 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:08:40.335878 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.335838 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:40.440378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.426712 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:08:40.499780 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.499739 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:40.609541 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.609447 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:08:40.633177 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.633131 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:40.678409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.678370 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:40.678728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.678695 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:40.711478 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.711441 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:40.711738 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.711713 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:40.837801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.837759 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:40.988284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:40.988236 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:41.011506 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:41.011464 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:41.293152 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:41.293052 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:41.423639 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:41.423584 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:41.452526 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:41.452468 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 03 19:08:41.499135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:41.499092 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:41.504774 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:41.503246 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:41.556241 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:08:41.548992 5019 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.285s" Nov 03 19:08:41.698201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:41.698162 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:41.699257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:41.699220 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:41.720307 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:41.720276 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:41.720611 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:41.720581 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:41.872415 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:41.848549 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:42.123488 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.108475 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:42.123488 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.108541 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:42.270359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.245366 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:42.315351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.294560 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:42.459545 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.430061 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:42.469168 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.469093 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:42.551498 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.549397 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:08:42.551498 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.549818 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:42.622000 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:08:42.621961 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:08:42.701353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.699353 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:42.701353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.699413 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:42.748410 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.744484 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:42.748410 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.744565 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:42.862354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.861941 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:42.878826 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.878706 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:42.878826 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.878765 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:43.002539 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:42.995080 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:43.119514 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.118421 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:43.354620 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.354475 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:43.430638 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.430229 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:43.485992 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.473494 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 03 19:08:43.517984 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.516692 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:08:43.526368 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.524177 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:08:43.551695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.550698 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:43.551695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.551060 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:08:43.712463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.683848 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:43.802432 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.715167 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:43.977416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.782752 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:43.977416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.949559 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:43.977416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.875469 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:43.977416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.782925 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:43.977416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.949949 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:43.977416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.879983 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:43.977416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.949996 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:43.977416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:43.951052 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:08:44.340408 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:44.339439 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:08:44.429668 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:44.428275 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:08:44.454873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:44.454836 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:08:44.737613 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:44.737578 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:44.738205 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:44.738174 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:44.738449 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:44.738311 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:08:44.738449 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:44.738386 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 19:08:44.915371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:44.884607 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:44.915371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:44.884668 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:45.628189 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:45.628150 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w"] Nov 03 19:08:45.628686 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:45.628642 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" podUID="45c8abc6-7671-4fc0-908a-c56b56f135f3" containerName="istio-proxy" containerID="cri-o://0a48ecb2c181d9f95a5b1a13a5e9997ab58893e54c282550c481748745c637cc" gracePeriod=30 Nov 03 19:08:45.679352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:45.669429 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:08:45.679352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:45.669493 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 19:08:45.896418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:45.893515 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:45.909355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:45.903961 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:45.909355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:45.904017 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:46.527071 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:46.527036 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:46.530475 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:46.530451 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:08:46.733257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:46.733205 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:46.735257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:46.735219 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:08:46.875632 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:46.875156 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:08:46.875632 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:46.875221 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 19:08:48.875252 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:48.875214 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:08:48.875767 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:48.875388 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:49.798843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:49.798805 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:49.800038 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:49.800015 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:08:50.901020 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:50.900939 5019 generic.go:356] "Generic (PLEG): container finished" podID="45c8abc6-7671-4fc0-908a-c56b56f135f3" containerID="0a48ecb2c181d9f95a5b1a13a5e9997ab58893e54c282550c481748745c637cc" exitCode=0 Nov 03 19:08:50.901423 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:50.901023 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" event={"ID":"45c8abc6-7671-4fc0-908a-c56b56f135f3","Type":"ContainerDied","Data":"0a48ecb2c181d9f95a5b1a13a5e9997ab58893e54c282550c481748745c637cc"} Nov 03 19:08:50.935992 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:50.935972 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:08:51.062790 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.062742 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-token\" (UniqueName: \"kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-token\") pod \"45c8abc6-7671-4fc0-908a-c56b56f135f3\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " Nov 03 19:08:51.063043 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.062956 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-kube-api-access-nvttn\" (UniqueName: \"kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-kube-api-access-nvttn\") pod \"45c8abc6-7671-4fc0-908a-c56b56f135f3\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " Nov 03 19:08:51.063043 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.063012 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-certs\") pod \"45c8abc6-7671-4fc0-908a-c56b56f135f3\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " Nov 03 19:08:51.063258 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.063052 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c8abc6-7671-4fc0-908a-c56b56f135f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c8abc6-7671-4fc0-908a-c56b56f135f3-istiod-ca-cert\") pod \"45c8abc6-7671-4fc0-908a-c56b56f135f3\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " Nov 03 19:08:51.063258 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.063181 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-socket\") pod \"45c8abc6-7671-4fc0-908a-c56b56f135f3\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " Nov 03 19:08:51.063258 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.063244 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-credential-socket\") pod \"45c8abc6-7671-4fc0-908a-c56b56f135f3\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " Nov 03 19:08:51.063482 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.063300 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-envoy\") pod \"45c8abc6-7671-4fc0-908a-c56b56f135f3\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " Nov 03 19:08:51.063482 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.063320 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-data\") pod \"45c8abc6-7671-4fc0-908a-c56b56f135f3\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " Nov 03 19:08:51.063482 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.063386 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-podinfo\") pod \"45c8abc6-7671-4fc0-908a-c56b56f135f3\" (UID: \"45c8abc6-7671-4fc0-908a-c56b56f135f3\") " Nov 03 19:08:51.064743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.064263 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-certs" pod "45c8abc6-7671-4fc0-908a-c56b56f135f3" (UID: "45c8abc6-7671-4fc0-908a-c56b56f135f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:51.064743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.064296 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-credential-socket" pod "45c8abc6-7671-4fc0-908a-c56b56f135f3" (UID: "45c8abc6-7671-4fc0-908a-c56b56f135f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:51.064743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.064462 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-data" pod "45c8abc6-7671-4fc0-908a-c56b56f135f3" (UID: "45c8abc6-7671-4fc0-908a-c56b56f135f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:51.064743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.064492 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-socket" pod "45c8abc6-7671-4fc0-908a-c56b56f135f3" (UID: "45c8abc6-7671-4fc0-908a-c56b56f135f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:51.064743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.064591 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c8abc6-7671-4fc0-908a-c56b56f135f3-istiod-ca-cert" pod "45c8abc6-7671-4fc0-908a-c56b56f135f3" (UID: "45c8abc6-7671-4fc0-908a-c56b56f135f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:51.066654 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.066634 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-kube-api-access-nvttn" pod "45c8abc6-7671-4fc0-908a-c56b56f135f3" (UID: "45c8abc6-7671-4fc0-908a-c56b56f135f3"). InnerVolumeSpecName "kube-api-access-nvttn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:51.067161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.067087 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-token" pod "45c8abc6-7671-4fc0-908a-c56b56f135f3" (UID: "45c8abc6-7671-4fc0-908a-c56b56f135f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:51.067276 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.067165 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-podinfo" pod "45c8abc6-7671-4fc0-908a-c56b56f135f3" (UID: "45c8abc6-7671-4fc0-908a-c56b56f135f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:51.067352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.067285 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-envoy" pod "45c8abc6-7671-4fc0-908a-c56b56f135f3" (UID: "45c8abc6-7671-4fc0-908a-c56b56f135f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:51.166501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.166417 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.166501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.166458 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.166501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.166475 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.166501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.166489 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.166741 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.166505 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvttn\" (UniqueName: \"kubernetes.io/projected/45c8abc6-7671-4fc0-908a-c56b56f135f3-kube-api-access-nvttn\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.166741 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.166520 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.166741 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.166533 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c8abc6-7671-4fc0-908a-c56b56f135f3-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.166741 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.166547 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.166741 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.166560 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c8abc6-7671-4fc0-908a-c56b56f135f3-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:51.910261 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.910231 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" Nov 03 19:08:51.910747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.910265 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w" event={"ID":"45c8abc6-7671-4fc0-908a-c56b56f135f3","Type":"ContainerDied","Data":"94035224675c925fa406f2adcfcab0c84b87125213619893524078546821a747"} Nov 03 19:08:51.910747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:51.910349 5019 scope.go:122] "RemoveContainer" containerID="0a48ecb2c181d9f95a5b1a13a5e9997ab58893e54c282550c481748745c637cc" Nov 03 19:08:52.114973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:52.114941 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w"] Nov 03 19:08:52.159595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:52.159566 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qn57w"] Nov 03 19:08:54.152023 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:08:54.151993 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45c8abc6-7671-4fc0-908a-c56b56f135f3" path="/var/lib/kubelet/pods/45c8abc6-7671-4fc0-908a-c56b56f135f3/volumes" Nov 03 19:09:00.167270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.167232 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf"] Nov 03 19:09:00.168223 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.168199 5019 cpu_manager.go:414] "containerMap: removing container" podUID="45c8abc6-7671-4fc0-908a-c56b56f135f3" containerName="istio-proxy" Nov 03 19:09:00.168223 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.168223 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c8abc6-7671-4fc0-908a-c56b56f135f3" containerName="istio-proxy" Nov 03 19:09:00.168465 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.168388 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c8abc6-7671-4fc0-908a-c56b56f135f3" containerName="istio-proxy" Nov 03 19:09:00.173948 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.173927 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.213589 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.213560 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52"] Nov 03 19:09:00.217842 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.217824 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.262935 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.262899 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf"] Nov 03 19:09:00.294537 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.294487 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52"] Nov 03 19:09:00.307026 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.306989 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.307206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307051 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.307206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307155 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44pww\" (UniqueName: \"kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-kube-api-access-44pww\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.307337 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307218 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.307337 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307313 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.307450 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307385 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.307502 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307465 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.307554 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307523 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.307636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307619 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eda232-a5c8-4016-866b-21c73206765a-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.307693 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307671 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.307751 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307701 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.307801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307752 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z72xp\" (UniqueName: \"kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-kube-api-access-z72xp\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.307852 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307795 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.307905 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307858 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.307959 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307915 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eda232-a5c8-4016-866b-21c73206765a-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.307959 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307940 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.308064 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.307977 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2a7348-14ea-449a-9988-99ff5f48506f-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.308064 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.308026 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.409615 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.409584 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eda232-a5c8-4016-866b-21c73206765a-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.409788 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.409631 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.409788 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.409720 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.409788 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.409759 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z72xp\" (UniqueName: \"kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-kube-api-access-z72xp\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.409940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.409785 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.410000 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.409946 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.410051 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410024 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eda232-a5c8-4016-866b-21c73206765a-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.410100 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410052 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.410100 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410076 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2a7348-14ea-449a-9988-99ff5f48506f-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.410203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410120 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.410203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410141 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.410203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410168 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.410393 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410218 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.410393 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410267 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44pww\" (UniqueName: \"kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-kube-api-access-44pww\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.410393 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410304 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.410556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410410 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.410556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410478 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.410556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410500 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.410701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410591 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.410701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410658 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.410701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410670 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.410863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410709 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.410863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410831 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.411026 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.410964 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.411151 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.411100 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.411363 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.411303 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2a7348-14ea-449a-9988-99ff5f48506f-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.411363 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.411313 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.411502 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.411466 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eda232-a5c8-4016-866b-21c73206765a-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.412742 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.412716 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.412856 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.412743 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eda232-a5c8-4016-866b-21c73206765a-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.413096 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.413079 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.413708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.413689 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.467927 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.467858 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.468113 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.468091 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44pww\" (UniqueName: \"kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-kube-api-access-44pww\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.468275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.468253 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.468537 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.468519 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z72xp\" (UniqueName: \"kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-kube-api-access-z72xp\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.488213 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.488178 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:00.532541 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.532511 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:00.917111 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:09:00.917076 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2a7348_14ea_449a_9988_99ff5f48506f.slice/crio-8b62c5d6ce914c7083ab1ac68fc63659e675f6fd815f3514b27dcadc71ddbe7f WatchSource:0}: Error finding container 8b62c5d6ce914c7083ab1ac68fc63659e675f6fd815f3514b27dcadc71ddbe7f: Status 404 returned error can't find the container with id 8b62c5d6ce914c7083ab1ac68fc63659e675f6fd815f3514b27dcadc71ddbe7f Nov 03 19:09:00.917267 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.917164 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf"] Nov 03 19:09:00.919926 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.919887 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:00.920042 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.919969 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:00.920042 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.920013 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:00.970742 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:09:00.970713 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44eda232_a5c8_4016_866b_21c73206765a.slice/crio-180e8c7bf5b947aa13ed3dd74d9c7a65f2a68657a3281500bd1c87a1f3b6eca4 WatchSource:0}: Error finding container 180e8c7bf5b947aa13ed3dd74d9c7a65f2a68657a3281500bd1c87a1f3b6eca4: Status 404 returned error can't find the container with id 180e8c7bf5b947aa13ed3dd74d9c7a65f2a68657a3281500bd1c87a1f3b6eca4 Nov 03 19:09:00.972122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.972098 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52"] Nov 03 19:09:00.973673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.973634 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:00.973757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.973696 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:00.973757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:00.973730 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:01.001970 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:01.001936 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" event={"ID":"44eda232-a5c8-4016-866b-21c73206765a","Type":"ContainerStarted","Data":"180e8c7bf5b947aa13ed3dd74d9c7a65f2a68657a3281500bd1c87a1f3b6eca4"} Nov 03 19:09:01.003660 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:01.003635 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" event={"ID":"7a2a7348-14ea-449a-9988-99ff5f48506f","Type":"ContainerStarted","Data":"57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9"} Nov 03 19:09:01.003756 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:01.003677 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" event={"ID":"7a2a7348-14ea-449a-9988-99ff5f48506f","Type":"ContainerStarted","Data":"8b62c5d6ce914c7083ab1ac68fc63659e675f6fd815f3514b27dcadc71ddbe7f"} Nov 03 19:09:01.117144 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:01.117096 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" podStartSLOduration=1.117082585 podStartE2EDuration="1.117082585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:01.115923826 +0000 UTC m=+1475.088485052" watchObservedRunningTime="2026-11-03 19:09:01.117082585 +0000 UTC m=+1475.089643819" Nov 03 19:09:01.491362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:01.488767 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:01.499471 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:01.495459 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:09:01.499471 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:01.495523 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" podUID="7a2a7348-14ea-449a-9988-99ff5f48506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 19:09:02.027304 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:02.027263 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" event={"ID":"44eda232-a5c8-4016-866b-21c73206765a","Type":"ContainerStarted","Data":"87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998"} Nov 03 19:09:02.158699 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:02.158647 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" podStartSLOduration=2.15863434 podStartE2EDuration="2.15863434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:02.152892992 +0000 UTC m=+1476.125454174" watchObservedRunningTime="2026-11-03 19:09:02.15863434 +0000 UTC m=+1476.131195604" Nov 03 19:09:02.494839 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:02.494797 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:09:02.495496 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:02.495457 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" podUID="7a2a7348-14ea-449a-9988-99ff5f48506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 19:09:02.533041 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:02.533011 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:02.534011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:02.533989 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:09:02.534192 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:02.534170 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" podUID="44eda232-a5c8-4016-866b-21c73206765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 19:09:03.506353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:03.506019 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:09:03.506353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:03.506079 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" podUID="7a2a7348-14ea-449a-9988-99ff5f48506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 19:09:03.532873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:03.532837 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:09:03.533043 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:03.532897 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" podUID="44eda232-a5c8-4016-866b-21c73206765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 19:09:04.505351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:04.492444 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:09:04.505351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:04.492508 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" podUID="7a2a7348-14ea-449a-9988-99ff5f48506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 19:09:04.534028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:04.533902 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:09:04.534028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:04.533967 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" podUID="44eda232-a5c8-4016-866b-21c73206765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 19:09:05.494352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:05.493140 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:09:05.494352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:05.493201 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" podUID="7a2a7348-14ea-449a-9988-99ff5f48506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 19:09:05.533094 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:05.533043 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:09:05.533415 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:05.533373 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" podUID="44eda232-a5c8-4016-866b-21c73206765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 19:09:06.489351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:06.488756 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:09:06.489351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:06.488838 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" podUID="7a2a7348-14ea-449a-9988-99ff5f48506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 19:09:06.534144 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:06.534104 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:09:06.534485 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:06.534430 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" podUID="44eda232-a5c8-4016-866b-21c73206765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 19:09:07.488808 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:07.488753 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:09:07.489154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:07.489100 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" podUID="7a2a7348-14ea-449a-9988-99ff5f48506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 19:09:07.534216 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:07.533775 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:09:07.534216 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:07.533850 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" podUID="44eda232-a5c8-4016-866b-21c73206765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 19:09:08.983301 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:08.981085 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm"] Nov 03 19:09:08.998648 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:08.998613 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.091355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.063840 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:09.091355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.079482 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:09.127405 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.106704 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:09:09.127405 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.106993 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-46ccd045-openshift-default-dockercfg-87s4t\"" Nov 03 19:09:09.228354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.219271 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-envoy\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.235586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.228792 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xm5b\" (UniqueName: \"kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-kube-api-access-7xm5b\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.260162 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.235972 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.500396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.495802 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-istio-token\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.546715 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.531843 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197e4439-81c3-493d-a04a-badc2299d2da-istio-podinfo\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.573351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.547009 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-credential-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.573678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.573646 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-certs\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.573842 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.573827 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-data\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.577501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.574150 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197e4439-81c3-493d-a04a-badc2299d2da-istiod-ca-cert\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.615942 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.587852 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm"] Nov 03 19:09:09.616226 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.588965 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:09.616430 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.616400 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" podUID="44eda232-a5c8-4016-866b-21c73206765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:09.709434 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.700708 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:09.709434 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.701872 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-envoy\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.709434 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.701938 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xm5b\" (UniqueName: \"kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-kube-api-access-7xm5b\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.709434 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.701972 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.709434 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.702100 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:09.748817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.702065 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-istio-token\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.748817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.744415 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197e4439-81c3-493d-a04a-badc2299d2da-istio-podinfo\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.748817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.744472 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-credential-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.748817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.744512 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-certs\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.748817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.744560 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-data\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.748817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.744777 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197e4439-81c3-493d-a04a-badc2299d2da-istiod-ca-cert\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.748817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.745757 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197e4439-81c3-493d-a04a-badc2299d2da-istiod-ca-cert\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.748817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.746028 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.788606 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.760689 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:09:09.796805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.763090 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-credential-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.800470 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.776790 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-certs\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.801438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.777058 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-data\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.804840 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.804796 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-envoy\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:09.917701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:09.900139 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xm5b\" (UniqueName: \"kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-kube-api-access-7xm5b\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:10.119464 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:10.119248 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197e4439-81c3-493d-a04a-badc2299d2da-istio-podinfo\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:10.124521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:10.124462 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-istio-token\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:10.136626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:10.136598 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:10.576119 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:10.573658 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm"] Nov 03 19:09:10.576119 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:10.575899 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:10.576119 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:10.575990 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:10.576119 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:10.576028 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:10.891079 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:10.890960 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" event={"ID":"197e4439-81c3-493d-a04a-badc2299d2da","Type":"ContainerStarted","Data":"2a9cc46f27f7153513a00b50962fb3d23f4918d9e11027be052714ad37aadb35"} Nov 03 19:09:10.891079 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:10.891010 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" event={"ID":"197e4439-81c3-493d-a04a-badc2299d2da","Type":"ContainerStarted","Data":"7acb88feb38c45b95fc31b8cecb4de6755259ded90214947768274a22144073a"} Nov 03 19:09:11.012236 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:11.012174 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" podStartSLOduration=3.012151886 podStartE2EDuration="3.012151886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:11.003851414 +0000 UTC m=+1484.976412624" watchObservedRunningTime="2026-11-03 19:09:11.012151886 +0000 UTC m=+1484.984713070" Nov 03 19:09:11.148401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:11.142816 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:11.178834 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:11.178521 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:11.341897 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:09:11.341866 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:09:11.898063 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:11.898023 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:11.899031 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:11.899005 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:09:17.140372 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:17.139310 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:09:17.345597 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:17.338163 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:09:17.446428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:17.392959 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:09:17.447390 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:17.447346 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:17.476020 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:17.475882 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:17.530374 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:17.515153 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:17.684408 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:17.675597 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded" Nov 03 19:09:18.149405 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:18.149123 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr"] Nov 03 19:09:18.149884 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:18.149497 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.411809 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:18.411706 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.528231 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:18.528186 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 19:09:18.551358 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:18.551300 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.566757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:18.566713 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:18.690602 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:18.690561 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:19.158946 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:19.158845 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:09:19.170370 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:19.159718 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:19.199168 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:19.174574 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:19.293353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:19.281644 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c2fc8901-openshift-default-dockercfg-xnw2s\"" Nov 03 19:09:19.426865 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:19.426805 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:09:19.624384 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:19.599616 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:19.624384 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:19.599897 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:19.681423 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:19.650036 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:19.737476 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:19.737426 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.019263 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.019073 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr"] Nov 03 19:09:20.019761 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.019741 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:09:20.019966 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.019940 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:09:20.020966 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.020938 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:09:20.027823 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.027805 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:09:20.058030 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.057992 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:09:20.110066 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.110017 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-socket\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.110366 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.110312 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvjk\" (UniqueName: \"kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-kube-api-access-kpvjk\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.110574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.110547 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-podinfo\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.110889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.110868 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-data\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.111137 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.111119 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-credential-socket\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.111379 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.111362 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7d9b60-de71-44e5-a7f4-4792f7681941-istiod-ca-cert\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.111540 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.111524 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-token\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.111782 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.111767 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-envoy\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.118808 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.118782 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-certs\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.234361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.227834 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:09:20.274696 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.274514 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-certs\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.275087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.275052 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-socket\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.275279 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.275261 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvjk\" (UniqueName: \"kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-kube-api-access-kpvjk\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.275436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.275418 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-podinfo\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.275760 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.275745 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-data\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.275945 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.275930 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-credential-socket\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.276113 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.276100 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7d9b60-de71-44e5-a7f4-4792f7681941-istiod-ca-cert\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.376196 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.332563 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-token\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.376196 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.332783 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-envoy\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.376196 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.312467 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-socket\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.376196 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.311292 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7d9b60-de71-44e5-a7f4-4792f7681941-istiod-ca-cert\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.376196 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.311940 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-certs\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.405425 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.405383 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-envoy\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.407403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.406118 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-credential-socket\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.447352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.444028 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:09:20.473354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.468985 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:20.473354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.469044 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" podUID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.506606 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.506566 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-data\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.608953 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.608825 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.610540 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.610508 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 19:09:20.705887 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.705846 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.773392 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.752487 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.870098 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.870002 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:20.870403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.870366 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:20.898461 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.898407 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvjk\" (UniqueName: \"kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-kube-api-access-kpvjk\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.901853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.901821 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-token\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:20.983657 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:20.975703 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-podinfo\") pod \"e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:21.113352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.111116 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:21.336542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.325882 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:09:21.450008 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.449712 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:09:21.491372 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.487795 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:21.491372 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.487857 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.628354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.615545 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:21.628354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.615654 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.628354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.617804 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:09:21.628695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.628627 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.707472 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.703900 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:21.707472 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.703975 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.730628 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.708247 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 03 19:09:21.798277 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.764776 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.921452 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.904447 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:21.921452 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.904524 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:21.961495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.958272 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:21.961495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:21.958364 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.279538 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:22.278390 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Nov 03 19:09:22.323238 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:22.319890 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:22.323238 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:22.319961 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.353435 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:22.353394 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:22.358406 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:22.353965 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.367674 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:22.364841 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.457123 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:22.457078 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:09:22.719178 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:22.686450 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.719178 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:22.687482 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 19:09:22.802351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:22.800131 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:22.802351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:22.800481 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 03 19:09:23.060924 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.060804 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.104565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.104499 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.537249 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.536461 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:23.537249 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.536540 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.567378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.560097 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:09:23.636357 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.625179 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.715519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.715462 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.716186 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.716149 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.817417 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.800404 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:09:23.839354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.820912 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.973386 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.833490 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:23.973386 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.969012 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:23.973386 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.932155 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:23.973386 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:23.969119 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:24.328685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:24.328641 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:24.328896 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:24.328722 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:24.672405 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:24.640605 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:09:24.731032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:24.641167 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:24.774557 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:24.774488 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:24.800626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:24.800580 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:24.821446 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:24.800512 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:25.206100 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:25.206039 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 03 19:09:25.314355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:25.306160 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded" Nov 03 19:09:25.790912 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:25.771957 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vspbc" podUID="fc9e70c5-7751-4377-aacf-3c8c8530bdc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:25.792206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:25.792166 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:09:25.798514 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:25.798479 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:09:26.137412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:25.998377 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:26.137412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:25.998721 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:26.137412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:26.061183 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 19:09:26.226748 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:09:26.219844 5019 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.891s" Nov 03 19:09:26.400147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:26.388705 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:09:26.578352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:26.563643 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:26.671447 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:26.578189 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Nov 03 19:09:27.040513 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:27.040466 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:09:27.136484 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:27.136427 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:09:27.137183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:27.137152 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 03 19:09:27.527008 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:27.159970 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:27.529999 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:27.529952 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:27.530254 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:27.530229 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:27.539971 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:27.239588 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Nov 03 19:09:27.604216 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:27.238881 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:09:27.637353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:27.634378 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:27.723250 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:27.219406 5019 trace.go:236] Trace[2069469228]: "Calculate volume metrics of workload-socket for pod openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" (03-Nov-2026 19:09:26.134) (total time: 1085ms): Nov 03 19:09:27.723250 ip-10-0-139-179 kubenswrapper[5019]: Trace[2069469228]: [1.085056375s] [1.085056375s] END Nov 03 19:09:27.999873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:27.999829 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:28.073925 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:28.073880 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:09:28.159730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:28.159361 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:28.300257 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:09:28.230421 5019 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.958s" Nov 03 19:09:28.542298 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:28.542253 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:28.618415 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:28.606064 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:28.644420 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:28.634427 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:09:28.677364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:28.634546 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:28.722495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:28.702761 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" event={"ID":"6d7d9b60-de71-44e5-a7f4-4792f7681941","Type":"ContainerStarted","Data":"21fdf5ae46f4fa36773d5c917cf9c88151ffc788e36fba40d48039f9f61bd2f3"} Nov 03 19:09:28.722873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:28.722837 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr"] Nov 03 19:09:29.197586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:29.186142 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 19:09:29.395704 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:29.395663 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:29.468236 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:29.467366 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:09:29.529937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:29.528359 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:09:29.529937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:29.528421 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:09:29.529937 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:09:29.528850 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2389b6ad_dc9d_46e4_9380_3f5595fe53be.slice/crio-fe3840178bc6e01bf774b768ba9bef30ab69285ef487fc8b7d1424b71fd2d714\": RecentStats: unable to find data in memory cache]" Nov 03 19:09:29.529937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:29.529314 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:09:29.532555 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:29.531925 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:09:29.532946 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:29.532925 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:09:29.551914 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:29.551886 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:09:29.558445 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:29.558425 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:09:30.529409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:30.529361 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" event={"ID":"6d7d9b60-de71-44e5-a7f4-4792f7681941","Type":"ContainerStarted","Data":"f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659"} Nov 03 19:09:30.647273 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:30.647212 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" podStartSLOduration=13.647196669 podStartE2EDuration="13.647196669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:30.642678886 +0000 UTC m=+1504.615240067" watchObservedRunningTime="2026-11-03 19:09:30.647196669 +0000 UTC m=+1504.619757850" Nov 03 19:09:31.112152 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:31.112121 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:31.117624 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:31.117598 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:31.536896 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:31.536859 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:31.537878 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:31.537857 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:09:38.746287 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:38.746250 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l"] Nov 03 19:09:38.752169 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:38.752149 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:38.833349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:38.826639 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7a8bab4a-openshift-default-dockercfg-mqm88\"" Nov 03 19:09:38.970350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:38.945218 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l"] Nov 03 19:09:39.027456 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:38.972618 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrjxn\" (UniqueName: \"kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-kube-api-access-hrjxn\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.059568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.059519 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea2f83c9-6151-4788-a105-eb94e0e25754-istiod-ca-cert\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.059849 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.059824 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-envoy\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.060011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.059989 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-credential-socket\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.060247 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.060226 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-socket\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.086981 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.086947 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-token\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.087297 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.087277 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-data\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.087498 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.087469 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-podinfo\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.090919 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.087813 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-certs\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.340471 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.340362 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-podinfo\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.341005 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.340966 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-certs\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.343231 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.343206 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrjxn\" (UniqueName: \"kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-kube-api-access-hrjxn\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.343641 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.343608 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea2f83c9-6151-4788-a105-eb94e0e25754-istiod-ca-cert\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.343826 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.343806 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-envoy\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.344016 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.343993 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-credential-socket\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.344205 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.344186 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-socket\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.347527 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.347502 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-token\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.347782 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.347768 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-data\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.348482 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.348462 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-data\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.350302 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.350268 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea2f83c9-6151-4788-a105-eb94e0e25754-istiod-ca-cert\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.356377 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.354519 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-certs\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.356831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.356691 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-credential-socket\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.357711 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.357686 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-socket\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.381554 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.381516 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-podinfo\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.471246 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.471095 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrjxn\" (UniqueName: \"kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-kube-api-access-hrjxn\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.477877 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.476792 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-token\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.482574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.482547 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-envoy\") pod \"e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:39.679909 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:39.679878 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:40.107763 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:09:40.107726 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2f83c9_6151_4788_a105_eb94e0e25754.slice/crio-7f309e931af8b5af7a5c358de338bdba0e4d745a6aa42c8805a5a695730def76 WatchSource:0}: Error finding container 7f309e931af8b5af7a5c358de338bdba0e4d745a6aa42c8805a5a695730def76: Status 404 returned error can't find the container with id 7f309e931af8b5af7a5c358de338bdba0e4d745a6aa42c8805a5a695730def76 Nov 03 19:09:40.108699 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:40.108672 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l"] Nov 03 19:09:40.110467 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:40.110432 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:40.110569 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:40.110512 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:40.110569 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:40.110557 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:09:40.688468 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:40.686315 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" event={"ID":"ea2f83c9-6151-4788-a105-eb94e0e25754","Type":"ContainerStarted","Data":"dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a"} Nov 03 19:09:40.688468 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:40.686409 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" event={"ID":"ea2f83c9-6151-4788-a105-eb94e0e25754","Type":"ContainerStarted","Data":"7f309e931af8b5af7a5c358de338bdba0e4d745a6aa42c8805a5a695730def76"} Nov 03 19:09:41.082403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:41.078187 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" podStartSLOduration=3.07816462 podStartE2EDuration="3.07816462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:40.873918862 +0000 UTC m=+1514.846480045" watchObservedRunningTime="2026-11-03 19:09:41.07816462 +0000 UTC m=+1515.050725806" Nov 03 19:09:41.686597 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:41.685962 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:41.688687 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:41.688656 5019 patch_prober.go:29] interesting pod/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:09:41.688918 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:41.688892 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 19:09:41.729073 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:09:41.729036 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:09:42.685856 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:42.685825 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:42.709747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:42.709713 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:42.710576 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:42.710550 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:09:54.616771 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:54.616671 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 19:09:54.967906 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:54.967859 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:55.045494 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:55.045438 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:09:55.248522 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:09:55.248418 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 19:10:11.303273 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:10:11.303228 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:10:21.510831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.510800 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl"] Nov 03 19:10:21.515759 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.515717 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:21.564889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.564859 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c2fc8901-serving-cert\"" Nov 03 19:10:21.592296 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.592245 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-maas-api-tls\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:21.592655 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.592628 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt58x\" (UniqueName: \"kubernetes.io/projected/f120c950-0666-425a-9e43-33dbeec944e3-kube-api-access-wt58x\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:21.592951 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.592925 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f120c950-0666-425a-9e43-33dbeec944e3-openshift-service-ca\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:21.593074 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.592978 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-metrics-certs\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:21.618088 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.618053 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl"] Nov 03 19:10:21.694545 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.694504 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-maas-api-tls\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:21.694744 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.694596 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt58x\" (UniqueName: \"kubernetes.io/projected/f120c950-0666-425a-9e43-33dbeec944e3-kube-api-access-wt58x\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:21.694744 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:10:21.694723 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-serving-cert: secret "maas-api-e2e-ait-c2fc8901-serving-cert" not found Nov 03 19:10:21.694884 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:10:21.694807 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-maas-api-tls podName:f120c950-0666-425a-9e43-33dbeec944e3 nodeName:}" failed. No retries permitted until 2026-11-03 19:10:22.194786121 +0000 UTC m=+1556.167347299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-maas-api-tls") pod "maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" (UID: "f120c950-0666-425a-9e43-33dbeec944e3") : secret "maas-api-e2e-ait-c2fc8901-serving-cert" not found Nov 03 19:10:21.695057 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.694728 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f120c950-0666-425a-9e43-33dbeec944e3-openshift-service-ca\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:21.695149 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.695085 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-metrics-certs\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:21.695565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.695539 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f120c950-0666-425a-9e43-33dbeec944e3-openshift-service-ca\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:21.708378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.698561 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-metrics-certs\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:21.789642 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:21.784848 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt58x\" (UniqueName: \"kubernetes.io/projected/f120c950-0666-425a-9e43-33dbeec944e3-kube-api-access-wt58x\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:22.231134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:22.231093 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-maas-api-tls\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:22.233710 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:22.233680 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-maas-api-tls\") pod \"maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:22.432106 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:22.432063 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:22.812870 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:10:22.812837 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf120c950_0666_425a_9e43_33dbeec944e3.slice/crio-039422c61cbe23b71aca8dfa8b1638c04cd787ac3f3ff6df2b948bd01e4c5cb7 WatchSource:0}: Error finding container 039422c61cbe23b71aca8dfa8b1638c04cd787ac3f3ff6df2b948bd01e4c5cb7: Status 404 returned error can't find the container with id 039422c61cbe23b71aca8dfa8b1638c04cd787ac3f3ff6df2b948bd01e4c5cb7 Nov 03 19:10:22.813290 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:22.813250 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl"] Nov 03 19:10:22.814627 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:22.814604 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:10:23.382338 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:23.382292 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" event={"ID":"f120c950-0666-425a-9e43-33dbeec944e3","Type":"ContainerStarted","Data":"039422c61cbe23b71aca8dfa8b1638c04cd787ac3f3ff6df2b948bd01e4c5cb7"} Nov 03 19:10:24.393016 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:24.392972 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" event={"ID":"f120c950-0666-425a-9e43-33dbeec944e3","Type":"ContainerStarted","Data":"b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c"} Nov 03 19:10:24.393600 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:24.393047 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:24.508009 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:24.507949 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" podStartSLOduration=2.628415581 podStartE2EDuration="3.507929898s" totalImagesPullingTime="879.514317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:24.502530506 +0000 UTC m=+1558.475091687" watchObservedRunningTime="2026-11-03 19:10:24.507929898 +0000 UTC m=+1558.480491081" Nov 03 19:10:30.410124 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:30.410095 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:10:35.630461 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.630365 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l"] Nov 03 19:10:35.636145 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.636124 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:35.719514 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.719470 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-metrics-certs\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:35.719715 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.719682 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7zd\" (UniqueName: \"kubernetes.io/projected/588748a9-9f92-4684-8c97-aff207248735-kube-api-access-7c7zd\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:35.719822 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.719802 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/588748a9-9f92-4684-8c97-aff207248735-openshift-service-ca\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:35.719895 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.719838 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-maas-api-tls\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:35.734455 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.734425 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7a8bab4a-serving-cert\"" Nov 03 19:10:35.739939 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.739908 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l"] Nov 03 19:10:35.821499 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.821466 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7zd\" (UniqueName: \"kubernetes.io/projected/588748a9-9f92-4684-8c97-aff207248735-kube-api-access-7c7zd\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:35.821754 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.821585 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/588748a9-9f92-4684-8c97-aff207248735-openshift-service-ca\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:35.821754 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.821610 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-maas-api-tls\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:35.821969 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.821813 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-metrics-certs\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:35.821969 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:10:35.821828 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-serving-cert: secret "maas-api-e2e-mt-7a8bab4a-serving-cert" not found Nov 03 19:10:35.821969 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:10:35.821909 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-maas-api-tls podName:588748a9-9f92-4684-8c97-aff207248735 nodeName:}" failed. No retries permitted until 2026-11-03 19:10:36.321873706 +0000 UTC m=+1570.294434889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-maas-api-tls") pod "maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" (UID: "588748a9-9f92-4684-8c97-aff207248735") : secret "maas-api-e2e-mt-7a8bab4a-serving-cert" not found Nov 03 19:10:35.822900 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.822809 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/588748a9-9f92-4684-8c97-aff207248735-openshift-service-ca\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:35.825064 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.825038 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-metrics-certs\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:35.896940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:35.896669 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7zd\" (UniqueName: \"kubernetes.io/projected/588748a9-9f92-4684-8c97-aff207248735-kube-api-access-7c7zd\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:36.339342 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:36.339292 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-maas-api-tls\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:36.362362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:36.361317 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-maas-api-tls\") pod \"maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:36.549972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:36.549938 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:36.992691 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:36.992659 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l"] Nov 03 19:10:36.996867 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:10:36.996840 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588748a9_9f92_4684_8c97_aff207248735.slice/crio-8e1ab9aa957a1ac377efee296f570108f8ba0a79eefa6f6009a74bca070fb4bb WatchSource:0}: Error finding container 8e1ab9aa957a1ac377efee296f570108f8ba0a79eefa6f6009a74bca070fb4bb: Status 404 returned error can't find the container with id 8e1ab9aa957a1ac377efee296f570108f8ba0a79eefa6f6009a74bca070fb4bb Nov 03 19:10:37.573347 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:37.573046 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" event={"ID":"588748a9-9f92-4684-8c97-aff207248735","Type":"ContainerStarted","Data":"8e1ab9aa957a1ac377efee296f570108f8ba0a79eefa6f6009a74bca070fb4bb"} Nov 03 19:10:38.610194 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:38.610149 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" event={"ID":"588748a9-9f92-4684-8c97-aff207248735","Type":"ContainerStarted","Data":"bbde1e8f525f9100b6ed158f222780f45228a10c65cc9f6705679ec310b7cbff"} Nov 03 19:10:38.610626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:38.610293 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:38.724535 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:38.724473 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" podStartSLOduration=2.816450535 podStartE2EDuration="3.72445462s" totalImagesPullingTime="908.004085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:38.722573042 +0000 UTC m=+1572.695134223" watchObservedRunningTime="2026-11-03 19:10:38.72445462 +0000 UTC m=+1572.697015804" Nov 03 19:10:41.308344 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:10:41.308294 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:10:42.941499 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:42.941454 5019 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:10:42.942003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:42.941519 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:10:43.683068 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:43.682025 5019 generic.go:356] "Generic (PLEG): container finished" podID="197e4439-81c3-493d-a04a-badc2299d2da" containerID="2a9cc46f27f7153513a00b50962fb3d23f4918d9e11027be052714ad37aadb35" exitCode=0 Nov 03 19:10:43.683068 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:43.682125 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" event={"ID":"197e4439-81c3-493d-a04a-badc2299d2da","Type":"ContainerDied","Data":"2a9cc46f27f7153513a00b50962fb3d23f4918d9e11027be052714ad37aadb35"} Nov 03 19:10:43.690681 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:43.690654 5019 scope.go:122] "RemoveContainer" containerID="2a9cc46f27f7153513a00b50962fb3d23f4918d9e11027be052714ad37aadb35" Nov 03 19:10:43.692978 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:43.692936 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:10:43.693087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:43.693035 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:10:43.693151 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:43.693082 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:10:44.627564 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:44.627533 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:10:44.695858 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:44.695815 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" event={"ID":"197e4439-81c3-493d-a04a-badc2299d2da","Type":"ContainerStarted","Data":"db5adc0a926a992b1ca803e919e44fd2539dc19ba04585d5e5754f7d7f7913bf"} Nov 03 19:10:45.137402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:45.137356 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:10:45.137596 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:45.137538 5019 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 19:10:45.137656 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:45.137595 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 19:10:46.137558 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.137524 5019 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 19:10:46.137951 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.137573 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 19:10:46.558071 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.558039 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m"] Nov 03 19:10:46.573623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.573593 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.647799 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.647758 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rngjm\" (UniqueName: \"kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-kube-api-access-rngjm\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.647981 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.647952 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.648059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.648020 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.648120 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.648070 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.648195 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.648176 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.648276 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.648260 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d590414-a23c-4b37-840b-47f06e7b83b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.648383 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.648340 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.648505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.648489 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.648570 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.648512 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.749873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.749836 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.749901 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.749938 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.749962 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.750023 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d590414-a23c-4b37-840b-47f06e7b83b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.750043 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750433 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.750160 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750433 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.750177 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750433 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.750247 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rngjm\" (UniqueName: \"kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-kube-api-access-rngjm\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750433 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.750412 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750648 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.750437 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750767 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.750744 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750835 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.750767 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.750969 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.750943 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d590414-a23c-4b37-840b-47f06e7b83b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.752833 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.752807 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.752940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.752902 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.809605 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.809515 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m"] Nov 03 19:10:46.827653 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.827620 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.830103 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.830073 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rngjm\" (UniqueName: \"kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-kube-api-access-rngjm\") pod \"maas-default-gateway-openshift-default-77f556d647-xj77m\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:46.925878 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:46.925851 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:47.138146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:47.138069 5019 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 19:10:47.138595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:47.138142 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 19:10:47.400522 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:10:47.400484 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d590414_a23c_4b37_840b_47f06e7b83b5.slice/crio-39912ea8b6078a26d0d8511855bc1965c3b881e95142e95f4d9de19831e85371 WatchSource:0}: Error finding container 39912ea8b6078a26d0d8511855bc1965c3b881e95142e95f4d9de19831e85371: Status 404 returned error can't find the container with id 39912ea8b6078a26d0d8511855bc1965c3b881e95142e95f4d9de19831e85371 Nov 03 19:10:47.403924 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:47.403742 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:10:47.404365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:47.404316 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:10:47.404539 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:47.404516 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:10:47.405686 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:47.405661 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m"] Nov 03 19:10:47.756261 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:47.756218 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" event={"ID":"6d590414-a23c-4b37-840b-47f06e7b83b5","Type":"ContainerStarted","Data":"caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c"} Nov 03 19:10:47.756473 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:47.756298 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" event={"ID":"6d590414-a23c-4b37-840b-47f06e7b83b5","Type":"ContainerStarted","Data":"39912ea8b6078a26d0d8511855bc1965c3b881e95142e95f4d9de19831e85371"} Nov 03 19:10:47.902351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:47.902048 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" podStartSLOduration=1.902026451 podStartE2EDuration="1.902026451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:47.876839376 +0000 UTC m=+1581.849400558" watchObservedRunningTime="2026-11-03 19:10:47.902026451 +0000 UTC m=+1581.874587634" Nov 03 19:10:47.925956 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:47.925923 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:48.137509 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:48.137429 5019 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 19:10:48.137509 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:48.137486 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 19:10:48.928065 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:48.928031 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xj77m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 19:10:48.928583 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:48.928091 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 19:10:49.137613 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:49.137579 5019 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 19:10:49.137779 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:49.137642 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 19:10:49.927354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:49.926729 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xj77m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 19:10:49.927354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:49.926792 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 19:10:50.136909 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:50.136872 5019 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 19:10:50.137300 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:50.136925 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 19:10:50.944426 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:50.939942 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xj77m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 19:10:50.944426 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:50.940009 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 19:10:51.142631 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:51.142490 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:10:51.143451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:51.143382 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:10:51.144505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:51.144485 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:10:51.926203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:51.926168 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xj77m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 19:10:51.926402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:51.926228 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 19:10:52.926794 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:52.926755 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xj77m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 19:10:52.927216 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:52.926818 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 19:10:53.942552 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:53.942500 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xj77m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 19:10:53.943225 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:53.943191 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 19:10:54.926510 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:54.926469 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xj77m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 19:10:54.926716 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:54.926555 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 19:10:55.926468 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:55.926418 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xj77m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 19:10:55.926935 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:55.926525 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 19:10:57.633736 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:57.633707 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:57.635023 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:57.635004 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:10:57.635678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:10:57.635653 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:11:02.579497 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:02.579459 5019 patch_prober.go:29] interesting pod/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:11:02.579897 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:02.579535 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:11:05.200429 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:05.200392 5019 generic.go:356] "Generic (PLEG): container finished" podID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerID="f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659" exitCode=0 Nov 03 19:11:05.201003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:05.200442 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" event={"ID":"6d7d9b60-de71-44e5-a7f4-4792f7681941","Type":"ContainerDied","Data":"f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659"} Nov 03 19:11:05.201003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:05.200932 5019 scope.go:122] "RemoveContainer" containerID="f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659" Nov 03 19:11:05.202692 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:05.202649 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:11:05.202840 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:05.202748 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:11:05.202840 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:05.202802 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:11:06.211741 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:06.211705 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" event={"ID":"6d7d9b60-de71-44e5-a7f4-4792f7681941","Type":"ContainerStarted","Data":"9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5"} Nov 03 19:11:07.111603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:07.111571 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:11:07.111927 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:07.111758 5019 patch_prober.go:29] interesting pod/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 19:11:07.111927 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:07.111814 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 19:11:08.112605 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:08.112573 5019 patch_prober.go:29] interesting pod/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 19:11:08.112993 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:08.112629 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 19:11:09.112172 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:09.112128 5019 patch_prober.go:29] interesting pod/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 19:11:09.112402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:09.112194 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 19:11:10.112705 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:10.112664 5019 patch_prober.go:29] interesting pod/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 19:11:10.113109 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:10.112750 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 19:11:11.112642 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:11.112607 5019 patch_prober.go:29] interesting pod/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 19:11:11.112824 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:11.112674 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 19:11:11.321367 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:11:11.321305 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:11:12.112587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:12.112544 5019 patch_prober.go:29] interesting pod/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 19:11:12.112790 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:12.112620 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 19:11:13.116833 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:13.116806 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:11:13.117248 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:13.117231 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:11:13.118008 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:13.117987 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:11:28.198004 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:28.195985 5019 patch_prober.go:29] interesting pod/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": read tcp 10.132.0.2:49114->10.132.0.106:15021: read: connection reset by peer" start-of-body= Nov 03 19:11:28.198004 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:28.196068 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": read tcp 10.132.0.2:49114->10.132.0.106:15021: read: connection reset by peer" Nov 03 19:11:28.560890 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:28.560803 5019 generic.go:356] "Generic (PLEG): container finished" podID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerID="dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a" exitCode=0 Nov 03 19:11:28.561044 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:28.560889 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" event={"ID":"ea2f83c9-6151-4788-a105-eb94e0e25754","Type":"ContainerDied","Data":"dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a"} Nov 03 19:11:28.561539 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:28.561518 5019 scope.go:122] "RemoveContainer" containerID="dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a" Nov 03 19:11:28.563768 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:28.563729 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:11:28.563871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:28.563822 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:11:28.563915 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:28.563869 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:11:29.570415 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:29.570375 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" event={"ID":"ea2f83c9-6151-4788-a105-eb94e0e25754","Type":"ContainerStarted","Data":"b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348"} Nov 03 19:11:29.684351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:29.683828 5019 patch_prober.go:29] interesting pod/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:11:29.684351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:29.683903 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 19:11:29.684626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:29.684377 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:11:30.684693 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:30.684657 5019 patch_prober.go:29] interesting pod/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:11:30.685307 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:30.685268 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 19:11:31.683352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:31.681733 5019 patch_prober.go:29] interesting pod/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:11:31.683352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:31.681804 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 19:11:32.795350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:32.764565 5019 patch_prober.go:29] interesting pod/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:11:32.795350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:32.764648 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 19:11:33.693354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:33.682437 5019 patch_prober.go:29] interesting pod/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:11:33.693354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:33.682518 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 19:11:34.681771 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:34.681736 5019 patch_prober.go:29] interesting pod/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:11:34.682315 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:34.682286 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 19:11:35.687073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:35.687035 5019 patch_prober.go:29] interesting pod/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:11:35.687632 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:35.687593 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 19:11:36.686662 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:36.686629 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:11:36.687055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:36.687023 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:11:36.688137 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:11:36.688108 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:11:41.590599 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:11:41.590557 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:12:03.025154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:03.025070 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:12:04.982605 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:04.982562 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm"] Nov 03 19:12:04.995971 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:04.995927 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" containerID="cri-o://db5adc0a926a992b1ca803e919e44fd2539dc19ba04585d5e5754f7d7f7913bf" gracePeriod=30 Nov 03 19:12:06.389558 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:06.384883 5019 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:12:06.390405 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:06.390370 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:12:06.755201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:06.755159 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:12:10.542677 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.542645 5019 generic.go:356] "Generic (PLEG): container finished" podID="197e4439-81c3-493d-a04a-badc2299d2da" containerID="db5adc0a926a992b1ca803e919e44fd2539dc19ba04585d5e5754f7d7f7913bf" exitCode=0 Nov 03 19:12:10.543087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.542718 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" event={"ID":"197e4439-81c3-493d-a04a-badc2299d2da","Type":"ContainerDied","Data":"db5adc0a926a992b1ca803e919e44fd2539dc19ba04585d5e5754f7d7f7913bf"} Nov 03 19:12:10.543087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.542783 5019 scope.go:122] "RemoveContainer" containerID="2a9cc46f27f7153513a00b50962fb3d23f4918d9e11027be052714ad37aadb35" Nov 03 19:12:10.815893 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.815830 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:12:10.909878 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.909851 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/197e4439-81c3-493d-a04a-badc2299d2da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197e4439-81c3-493d-a04a-badc2299d2da-istio-podinfo\") pod \"197e4439-81c3-493d-a04a-badc2299d2da\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " Nov 03 19:12:10.910055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.909945 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-kube-api-access-7xm5b\" (UniqueName: \"kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-kube-api-access-7xm5b\") pod \"197e4439-81c3-493d-a04a-badc2299d2da\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " Nov 03 19:12:10.910055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.909971 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-socket\") pod \"197e4439-81c3-493d-a04a-badc2299d2da\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " Nov 03 19:12:10.910055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.910013 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-data\") pod \"197e4439-81c3-493d-a04a-badc2299d2da\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " Nov 03 19:12:10.910055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.910041 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/197e4439-81c3-493d-a04a-badc2299d2da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197e4439-81c3-493d-a04a-badc2299d2da-istiod-ca-cert\") pod \"197e4439-81c3-493d-a04a-badc2299d2da\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " Nov 03 19:12:10.910272 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.910148 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-istio-token\" (UniqueName: \"kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-istio-token\") pod \"197e4439-81c3-493d-a04a-badc2299d2da\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " Nov 03 19:12:10.910272 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.910256 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-socket" pod "197e4439-81c3-493d-a04a-badc2299d2da" (UID: "197e4439-81c3-493d-a04a-badc2299d2da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:10.910400 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.910281 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-credential-socket\") pod \"197e4439-81c3-493d-a04a-badc2299d2da\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " Nov 03 19:12:10.910400 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.910315 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-data" pod "197e4439-81c3-493d-a04a-badc2299d2da" (UID: "197e4439-81c3-493d-a04a-badc2299d2da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:10.910400 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.910362 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-certs\") pod \"197e4439-81c3-493d-a04a-badc2299d2da\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " Nov 03 19:12:10.910565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.910409 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-envoy\") pod \"197e4439-81c3-493d-a04a-badc2299d2da\" (UID: \"197e4439-81c3-493d-a04a-badc2299d2da\") " Nov 03 19:12:10.910565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.910541 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-credential-socket" pod "197e4439-81c3-493d-a04a-badc2299d2da" (UID: "197e4439-81c3-493d-a04a-badc2299d2da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:10.910669 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.910571 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197e4439-81c3-493d-a04a-badc2299d2da-istiod-ca-cert" pod "197e4439-81c3-493d-a04a-badc2299d2da" (UID: "197e4439-81c3-493d-a04a-badc2299d2da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:12:10.910669 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.910645 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-certs" pod "197e4439-81c3-493d-a04a-badc2299d2da" (UID: "197e4439-81c3-493d-a04a-badc2299d2da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:10.911549 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.911523 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.911677 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.911553 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.911677 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.911567 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197e4439-81c3-493d-a04a-badc2299d2da-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.911677 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.911580 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.911677 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.911593 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:10.912223 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.912201 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/197e4439-81c3-493d-a04a-badc2299d2da-istio-podinfo" pod "197e4439-81c3-493d-a04a-badc2299d2da" (UID: "197e4439-81c3-493d-a04a-badc2299d2da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:12:10.912453 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.912434 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-kube-api-access-7xm5b" pod "197e4439-81c3-493d-a04a-badc2299d2da" (UID: "197e4439-81c3-493d-a04a-badc2299d2da"). InnerVolumeSpecName "kube-api-access-7xm5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:10.912560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.912541 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-envoy" pod "197e4439-81c3-493d-a04a-badc2299d2da" (UID: "197e4439-81c3-493d-a04a-badc2299d2da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:10.912646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:10.912628 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-istio-token" pod "197e4439-81c3-493d-a04a-badc2299d2da" (UID: "197e4439-81c3-493d-a04a-badc2299d2da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:11.013031 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:11.013000 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197e4439-81c3-493d-a04a-badc2299d2da-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:11.013031 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:11.013028 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197e4439-81c3-493d-a04a-badc2299d2da-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:11.013245 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:11.013043 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xm5b\" (UniqueName: \"kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-kube-api-access-7xm5b\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:11.013245 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:11.013059 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197e4439-81c3-493d-a04a-badc2299d2da-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:11.300670 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:12:11.300635 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:12:11.565204 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:11.565105 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" event={"ID":"197e4439-81c3-493d-a04a-badc2299d2da","Type":"ContainerDied","Data":"7acb88feb38c45b95fc31b8cecb4de6755259ded90214947768274a22144073a"} Nov 03 19:12:11.565204 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:11.565126 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm" Nov 03 19:12:11.565204 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:11.565190 5019 scope.go:122] "RemoveContainer" containerID="db5adc0a926a992b1ca803e919e44fd2539dc19ba04585d5e5754f7d7f7913bf" Nov 03 19:12:11.809388 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:11.805591 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm"] Nov 03 19:12:11.810187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:11.810160 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-9grgm"] Nov 03 19:12:12.228648 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:12.228609 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197e4439-81c3-493d-a04a-badc2299d2da" path="/var/lib/kubelet/pods/197e4439-81c3-493d-a04a-badc2299d2da/volumes" Nov 03 19:12:16.224451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.224415 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl"] Nov 03 19:12:16.225044 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.224680 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" podUID="f120c950-0666-425a-9e43-33dbeec944e3" containerName="maas-api" containerID="cri-o://b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c" gracePeriod=30 Nov 03 19:12:16.678351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.664741 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:12:16.740353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.728892 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-metrics-certs\") pod \"f120c950-0666-425a-9e43-33dbeec944e3\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " Nov 03 19:12:16.740353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.729043 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-maas-api-tls\") pod \"f120c950-0666-425a-9e43-33dbeec944e3\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " Nov 03 19:12:16.740353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.729105 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f120c950-0666-425a-9e43-33dbeec944e3-kube-api-access-wt58x\" (UniqueName: \"kubernetes.io/projected/f120c950-0666-425a-9e43-33dbeec944e3-kube-api-access-wt58x\") pod \"f120c950-0666-425a-9e43-33dbeec944e3\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " Nov 03 19:12:16.740353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.729345 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f120c950-0666-425a-9e43-33dbeec944e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f120c950-0666-425a-9e43-33dbeec944e3-openshift-service-ca\") pod \"f120c950-0666-425a-9e43-33dbeec944e3\" (UID: \"f120c950-0666-425a-9e43-33dbeec944e3\") " Nov 03 19:12:16.742250 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.740934 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f120c950-0666-425a-9e43-33dbeec944e3-openshift-service-ca" pod "f120c950-0666-425a-9e43-33dbeec944e3" (UID: "f120c950-0666-425a-9e43-33dbeec944e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:12:16.801572 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.792024 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-metrics-certs" pod "f120c950-0666-425a-9e43-33dbeec944e3" (UID: "f120c950-0666-425a-9e43-33dbeec944e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:12:16.827396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.807596 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-maas-api-tls" pod "f120c950-0666-425a-9e43-33dbeec944e3" (UID: "f120c950-0666-425a-9e43-33dbeec944e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:12:16.827396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.808213 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f120c950-0666-425a-9e43-33dbeec944e3-kube-api-access-wt58x" pod "f120c950-0666-425a-9e43-33dbeec944e3" (UID: "f120c950-0666-425a-9e43-33dbeec944e3"). InnerVolumeSpecName "kube-api-access-wt58x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:16.836352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.832356 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt58x\" (UniqueName: \"kubernetes.io/projected/f120c950-0666-425a-9e43-33dbeec944e3-kube-api-access-wt58x\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:16.836352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.832459 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f120c950-0666-425a-9e43-33dbeec944e3-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:16.836352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.832477 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:16.836352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.832500 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f120c950-0666-425a-9e43-33dbeec944e3-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:16.944430 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.939269 5019 generic.go:356] "Generic (PLEG): container finished" podID="f120c950-0666-425a-9e43-33dbeec944e3" containerID="b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c" exitCode=0 Nov 03 19:12:16.944430 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.939385 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" event={"ID":"f120c950-0666-425a-9e43-33dbeec944e3","Type":"ContainerDied","Data":"b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c"} Nov 03 19:12:16.944430 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.939444 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" event={"ID":"f120c950-0666-425a-9e43-33dbeec944e3","Type":"ContainerDied","Data":"039422c61cbe23b71aca8dfa8b1638c04cd787ac3f3ff6df2b948bd01e4c5cb7"} Nov 03 19:12:16.944430 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.939477 5019 scope.go:122] "RemoveContainer" containerID="b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c" Nov 03 19:12:16.944430 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:16.939642 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl" Nov 03 19:12:17.029201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:17.028869 5019 scope.go:122] "RemoveContainer" containerID="b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c" Nov 03 19:12:17.029937 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:12:17.029832 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c\": container with ID starting with b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c not found: ID does not exist" containerID="b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c" Nov 03 19:12:17.029937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:17.029872 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c"} err="failed to get container status \"b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c\": rpc error: code = NotFound desc = could not find container \"b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c\": container with ID starting with b5b21242f420c54422298e255a9071eecace7027e586aa28224183d6fa1ffe4c not found: ID does not exist" Nov 03 19:12:17.139450 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:17.138490 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl"] Nov 03 19:12:17.142304 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:17.142270 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c2fc8901-76f848857d-s5vxl"] Nov 03 19:12:18.153427 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:18.153388 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f120c950-0666-425a-9e43-33dbeec944e3" path="/var/lib/kubelet/pods/f120c950-0666-425a-9e43-33dbeec944e3/volumes" Nov 03 19:12:20.188143 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:20.188109 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr"] Nov 03 19:12:20.188636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:20.188378 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" containerID="cri-o://9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5" gracePeriod=30 Nov 03 19:12:25.499779 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.499752 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:12:25.611884 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.611848 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-data\") pod \"6d7d9b60-de71-44e5-a7f4-4792f7681941\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " Nov 03 19:12:25.612062 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.611917 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-token\" (UniqueName: \"kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-token\") pod \"6d7d9b60-de71-44e5-a7f4-4792f7681941\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " Nov 03 19:12:25.612062 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.612009 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-kube-api-access-kpvjk\" (UniqueName: \"kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-kube-api-access-kpvjk\") pod \"6d7d9b60-de71-44e5-a7f4-4792f7681941\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " Nov 03 19:12:25.612154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.612090 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d7d9b60-de71-44e5-a7f4-4792f7681941-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7d9b60-de71-44e5-a7f4-4792f7681941-istiod-ca-cert\") pod \"6d7d9b60-de71-44e5-a7f4-4792f7681941\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " Nov 03 19:12:25.612154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.612131 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-envoy\") pod \"6d7d9b60-de71-44e5-a7f4-4792f7681941\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " Nov 03 19:12:25.612226 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.612162 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-data" pod "6d7d9b60-de71-44e5-a7f4-4792f7681941" (UID: "6d7d9b60-de71-44e5-a7f4-4792f7681941"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:25.612676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.612301 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-podinfo\") pod \"6d7d9b60-de71-44e5-a7f4-4792f7681941\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " Nov 03 19:12:25.612676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.612365 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-credential-socket\") pod \"6d7d9b60-de71-44e5-a7f4-4792f7681941\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " Nov 03 19:12:25.612676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.612418 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-certs\") pod \"6d7d9b60-de71-44e5-a7f4-4792f7681941\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " Nov 03 19:12:25.612676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.612526 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-socket\") pod \"6d7d9b60-de71-44e5-a7f4-4792f7681941\" (UID: \"6d7d9b60-de71-44e5-a7f4-4792f7681941\") " Nov 03 19:12:25.612947 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.612698 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7d9b60-de71-44e5-a7f4-4792f7681941-istiod-ca-cert" pod "6d7d9b60-de71-44e5-a7f4-4792f7681941" (UID: "6d7d9b60-de71-44e5-a7f4-4792f7681941"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:12:25.613101 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.613071 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-credential-socket" pod "6d7d9b60-de71-44e5-a7f4-4792f7681941" (UID: "6d7d9b60-de71-44e5-a7f4-4792f7681941"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:25.613284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.613264 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-certs" pod "6d7d9b60-de71-44e5-a7f4-4792f7681941" (UID: "6d7d9b60-de71-44e5-a7f4-4792f7681941"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:25.613560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.613453 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7d9b60-de71-44e5-a7f4-4792f7681941-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:25.613560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.613506 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:25.613560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.613521 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:25.613560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.613536 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:25.613560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.613533 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-socket" pod "6d7d9b60-de71-44e5-a7f4-4792f7681941" (UID: "6d7d9b60-de71-44e5-a7f4-4792f7681941"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:25.614863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.614831 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-kube-api-access-kpvjk" pod "6d7d9b60-de71-44e5-a7f4-4792f7681941" (UID: "6d7d9b60-de71-44e5-a7f4-4792f7681941"). InnerVolumeSpecName "kube-api-access-kpvjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:25.615232 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.615207 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-podinfo" pod "6d7d9b60-de71-44e5-a7f4-4792f7681941" (UID: "6d7d9b60-de71-44e5-a7f4-4792f7681941"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:12:25.615536 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.615516 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-token" pod "6d7d9b60-de71-44e5-a7f4-4792f7681941" (UID: "6d7d9b60-de71-44e5-a7f4-4792f7681941"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:25.615756 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.615733 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-envoy" pod "6d7d9b60-de71-44e5-a7f4-4792f7681941" (UID: "6d7d9b60-de71-44e5-a7f4-4792f7681941"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:25.715052 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.715015 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:25.715052 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.715046 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpvjk\" (UniqueName: \"kubernetes.io/projected/6d7d9b60-de71-44e5-a7f4-4792f7681941-kube-api-access-kpvjk\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:25.715250 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.715062 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:25.715250 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.715074 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7d9b60-de71-44e5-a7f4-4792f7681941-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:25.715250 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:25.715087 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7d9b60-de71-44e5-a7f4-4792f7681941-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:26.020547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.020515 5019 generic.go:356] "Generic (PLEG): container finished" podID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerID="9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5" exitCode=0 Nov 03 19:12:26.020721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.020569 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" Nov 03 19:12:26.020721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.020609 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" event={"ID":"6d7d9b60-de71-44e5-a7f4-4792f7681941","Type":"ContainerDied","Data":"9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5"} Nov 03 19:12:26.020721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.020667 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr" event={"ID":"6d7d9b60-de71-44e5-a7f4-4792f7681941","Type":"ContainerDied","Data":"21fdf5ae46f4fa36773d5c917cf9c88151ffc788e36fba40d48039f9f61bd2f3"} Nov 03 19:12:26.020721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.020692 5019 scope.go:122] "RemoveContainer" containerID="9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5" Nov 03 19:12:26.031863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.031842 5019 scope.go:122] "RemoveContainer" containerID="f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659" Nov 03 19:12:26.041463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.041443 5019 scope.go:122] "RemoveContainer" containerID="9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5" Nov 03 19:12:26.041713 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:12:26.041694 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5\": container with ID starting with 9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5 not found: ID does not exist" containerID="9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5" Nov 03 19:12:26.041775 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.041723 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5"} err="failed to get container status \"9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5\": rpc error: code = NotFound desc = could not find container \"9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5\": container with ID starting with 9bed1cbc0598246b6123849a96bcb8f5cb3e86afad9c0d799da77f15adca68f5 not found: ID does not exist" Nov 03 19:12:26.041775 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.041740 5019 scope.go:122] "RemoveContainer" containerID="f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659" Nov 03 19:12:26.041936 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:12:26.041923 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659\": container with ID starting with f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659 not found: ID does not exist" containerID="f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659" Nov 03 19:12:26.041975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.041939 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659"} err="failed to get container status \"f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659\": rpc error: code = NotFound desc = could not find container \"f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659\": container with ID starting with f9c07afb9e00de8cd56018206e06bc97d3dc2e881a1c713f747140cb00cdf659 not found: ID does not exist" Nov 03 19:12:26.184512 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.184475 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr"] Nov 03 19:12:26.191182 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.191154 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c2fc8901-openshift-default-fcc96c57-nbzrr"] Nov 03 19:12:26.864155 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.864123 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np"] Nov 03 19:12:26.865305 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865281 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f120c950-0666-425a-9e43-33dbeec944e3" containerName="maas-api" Nov 03 19:12:26.865441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865306 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f120c950-0666-425a-9e43-33dbeec944e3" containerName="maas-api" Nov 03 19:12:26.865441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865361 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" Nov 03 19:12:26.865441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865370 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" Nov 03 19:12:26.865441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865419 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" Nov 03 19:12:26.865441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865427 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" Nov 03 19:12:26.865592 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865443 5019 cpu_manager.go:414] "containerMap: removing container" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" Nov 03 19:12:26.865592 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865450 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" Nov 03 19:12:26.865651 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865634 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" Nov 03 19:12:26.865651 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865645 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" Nov 03 19:12:26.865711 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865660 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" Nov 03 19:12:26.865711 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865672 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f120c950-0666-425a-9e43-33dbeec944e3" containerName="maas-api" Nov 03 19:12:26.865711 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.865685 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" containerName="istio-proxy" Nov 03 19:12:26.898485 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.872586 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:26.945431 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.945395 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istiod-ca-cert\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:26.945820 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.945782 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-envoy\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:26.945976 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.945961 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-podinfo\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:26.946154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.946126 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ds4\" (UniqueName: \"kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-kube-api-access-t7ds4\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:26.946414 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.946394 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-token\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:26.946605 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.946578 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-data\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:26.946804 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.946788 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-certs\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:26.946963 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.946946 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-credential-socket\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:26.947148 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.947132 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-socket\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:26.959365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:26.952725 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d86d1cd0-openshift-default-dockercfg-hhczd\"" Nov 03 19:12:27.071566 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.069950 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istiod-ca-cert\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.071566 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.070084 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-envoy\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.071566 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.070108 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-podinfo\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.071566 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.070164 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ds4\" (UniqueName: \"kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-kube-api-access-t7ds4\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.215731 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.215695 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-token\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.216018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.215993 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-data\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.216263 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.216244 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-certs\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.216449 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.216430 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-credential-socket\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.216907 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.216605 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-socket\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.240255 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.240216 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-socket\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.244657 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.241115 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istiod-ca-cert\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.252471 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.252424 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-data\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.256002 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.255968 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-certs\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.357343 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.256673 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-credential-socket\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.405472 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.327919 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ds4\" (UniqueName: \"kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-kube-api-access-t7ds4\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.405716 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.352611 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-token\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.449696 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.449017 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-podinfo\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.469410 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.461629 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np"] Nov 03 19:12:27.483727 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.483682 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-envoy\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.648206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.648158 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:27.844871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.837970 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l"] Nov 03 19:12:27.845919 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:27.845864 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" podUID="588748a9-9f92-4684-8c97-aff207248735" containerName="maas-api" containerID="cri-o://bbde1e8f525f9100b6ed158f222780f45228a10c65cc9f6705679ec310b7cbff" gracePeriod=30 Nov 03 19:12:28.435632 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.428729 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d7d9b60-de71-44e5-a7f4-4792f7681941" path="/var/lib/kubelet/pods/6d7d9b60-de71-44e5-a7f4-4792f7681941/volumes" Nov 03 19:12:28.532841 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.532137 5019 generic.go:356] "Generic (PLEG): container finished" podID="588748a9-9f92-4684-8c97-aff207248735" containerID="bbde1e8f525f9100b6ed158f222780f45228a10c65cc9f6705679ec310b7cbff" exitCode=0 Nov 03 19:12:28.532841 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.532191 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" event={"ID":"588748a9-9f92-4684-8c97-aff207248735","Type":"ContainerDied","Data":"bbde1e8f525f9100b6ed158f222780f45228a10c65cc9f6705679ec310b7cbff"} Nov 03 19:12:28.657208 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.657180 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:12:28.661995 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:12:28.661962 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39acdc92_a6a2_488c_b2c2_6a6133f0ce38.slice/crio-68026dcf4c673c4e95da6fb1b0e17c8207e1de6fcc3fe1ace1091daf364592b0 WatchSource:0}: Error finding container 68026dcf4c673c4e95da6fb1b0e17c8207e1de6fcc3fe1ace1091daf364592b0: Status 404 returned error can't find the container with id 68026dcf4c673c4e95da6fb1b0e17c8207e1de6fcc3fe1ace1091daf364592b0 Nov 03 19:12:28.665080 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.665043 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:12:28.665192 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.665123 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:12:28.665192 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.665168 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:12:28.667442 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.667422 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np"] Nov 03 19:12:28.829353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.822153 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/588748a9-9f92-4684-8c97-aff207248735-kube-api-access-7c7zd\" (UniqueName: \"kubernetes.io/projected/588748a9-9f92-4684-8c97-aff207248735-kube-api-access-7c7zd\") pod \"588748a9-9f92-4684-8c97-aff207248735\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " Nov 03 19:12:28.829353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.822741 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/588748a9-9f92-4684-8c97-aff207248735-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/588748a9-9f92-4684-8c97-aff207248735-openshift-service-ca\") pod \"588748a9-9f92-4684-8c97-aff207248735\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " Nov 03 19:12:28.859283 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.856100 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-maas-api-tls\") pod \"588748a9-9f92-4684-8c97-aff207248735\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " Nov 03 19:12:28.859283 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.856297 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-metrics-certs\" (UniqueName: \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-metrics-certs\") pod \"588748a9-9f92-4684-8c97-aff207248735\" (UID: \"588748a9-9f92-4684-8c97-aff207248735\") " Nov 03 19:12:28.930350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.921850 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-metrics-certs" pod "588748a9-9f92-4684-8c97-aff207248735" (UID: "588748a9-9f92-4684-8c97-aff207248735"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:12:28.968412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:28.967061 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588748a9-9f92-4684-8c97-aff207248735-openshift-service-ca" pod "588748a9-9f92-4684-8c97-aff207248735" (UID: "588748a9-9f92-4684-8c97-aff207248735"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:12:29.052558 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.052488 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:29.052558 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.052526 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/588748a9-9f92-4684-8c97-aff207248735-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:29.088351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.069566 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-maas-api-tls" pod "588748a9-9f92-4684-8c97-aff207248735" (UID: "588748a9-9f92-4684-8c97-aff207248735"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:12:29.128531 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.126837 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588748a9-9f92-4684-8c97-aff207248735-kube-api-access-7c7zd" pod "588748a9-9f92-4684-8c97-aff207248735" (UID: "588748a9-9f92-4684-8c97-aff207248735"). InnerVolumeSpecName "kube-api-access-7c7zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:29.184936 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.184899 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c7zd\" (UniqueName: \"kubernetes.io/projected/588748a9-9f92-4684-8c97-aff207248735-kube-api-access-7c7zd\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:29.184936 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.184934 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588748a9-9f92-4684-8c97-aff207248735-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:29.542474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.542372 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" event={"ID":"39acdc92-a6a2-488c-b2c2-6a6133f0ce38","Type":"ContainerStarted","Data":"3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94"} Nov 03 19:12:29.542474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.542444 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" event={"ID":"39acdc92-a6a2-488c-b2c2-6a6133f0ce38","Type":"ContainerStarted","Data":"68026dcf4c673c4e95da6fb1b0e17c8207e1de6fcc3fe1ace1091daf364592b0"} Nov 03 19:12:29.544134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.544111 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" Nov 03 19:12:29.544276 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.544158 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l" event={"ID":"588748a9-9f92-4684-8c97-aff207248735","Type":"ContainerDied","Data":"8e1ab9aa957a1ac377efee296f570108f8ba0a79eefa6f6009a74bca070fb4bb"} Nov 03 19:12:29.544276 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.544239 5019 scope.go:122] "RemoveContainer" containerID="bbde1e8f525f9100b6ed158f222780f45228a10c65cc9f6705679ec310b7cbff" Nov 03 19:12:29.562906 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.562876 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm"] Nov 03 19:12:29.563687 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.563668 5019 cpu_manager.go:414] "containerMap: removing container" podUID="588748a9-9f92-4684-8c97-aff207248735" containerName="maas-api" Nov 03 19:12:29.563687 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.563687 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="588748a9-9f92-4684-8c97-aff207248735" containerName="maas-api" Nov 03 19:12:29.563839 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.563747 5019 cpu_manager.go:414] "containerMap: removing container" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" Nov 03 19:12:29.563839 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.563757 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197e4439-81c3-493d-a04a-badc2299d2da" containerName="istio-proxy" Nov 03 19:12:29.563956 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.563942 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="588748a9-9f92-4684-8c97-aff207248735" containerName="maas-api" Nov 03 19:12:29.569691 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.569671 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:29.620797 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.620767 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-144890-serving-cert\"" Nov 03 19:12:29.627110 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.627081 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm"] Nov 03 19:12:29.657231 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.657170 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" podStartSLOduration=3.657153355 podStartE2EDuration="3.657153355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:29.655253894 +0000 UTC m=+1683.627815077" watchObservedRunningTime="2026-11-03 19:12:29.657153355 +0000 UTC m=+1683.629714537" Nov 03 19:12:29.681895 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.680938 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:29.697024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.696993 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:29.697378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.697354 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:29.697617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.697589 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvfd\" (UniqueName: \"kubernetes.io/projected/313d392e-831e-428c-ad0d-0bbd03674212-kube-api-access-grvfd\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:29.697761 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.697746 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/313d392e-831e-428c-ad0d-0bbd03674212-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:29.767862 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.764158 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l"] Nov 03 19:12:29.769068 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.769040 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a8bab4a-6bbfcf8867-p6r2l"] Nov 03 19:12:29.826401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.813376 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:29.826401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.813546 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:29.826401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.813666 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvfd\" (UniqueName: \"kubernetes.io/projected/313d392e-831e-428c-ad0d-0bbd03674212-kube-api-access-grvfd\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:29.826401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.813801 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/313d392e-831e-428c-ad0d-0bbd03674212-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:29.826401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.814717 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/313d392e-831e-428c-ad0d-0bbd03674212-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:29.828643 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:12:29.828617 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-serving-cert: secret "maas-api-e2e-shared-a-w4-144890-serving-cert" not found Nov 03 19:12:29.828977 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:12:29.828924 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-maas-api-tls podName:313d392e-831e-428c-ad0d-0bbd03674212 nodeName:}" failed. No retries permitted until 2026-11-03 19:12:30.328837554 +0000 UTC m=+1684.301398728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-maas-api-tls") pod "maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" (UID: "313d392e-831e-428c-ad0d-0bbd03674212") : secret "maas-api-e2e-shared-a-w4-144890-serving-cert" not found Nov 03 19:12:29.890354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.889527 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:29.931352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:29.928906 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvfd\" (UniqueName: \"kubernetes.io/projected/313d392e-831e-428c-ad0d-0bbd03674212-kube-api-access-grvfd\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:30.232683 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:30.231509 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="588748a9-9f92-4684-8c97-aff207248735" path="/var/lib/kubelet/pods/588748a9-9f92-4684-8c97-aff207248735/volumes" Nov 03 19:12:30.361564 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:30.361530 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:30.363942 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:30.363916 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:30.485257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:30.485174 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:30.768418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:30.761616 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:31.295559 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:31.295527 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm"] Nov 03 19:12:31.295962 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:12:31.295937 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313d392e_831e_428c_ad0d_0bbd03674212.slice/crio-6f7b3b53484d2caa4edbaa5911b59031a44359c39ef4ad3f6fb3fe17d29ef832 WatchSource:0}: Error finding container 6f7b3b53484d2caa4edbaa5911b59031a44359c39ef4ad3f6fb3fe17d29ef832: Status 404 returned error can't find the container with id 6f7b3b53484d2caa4edbaa5911b59031a44359c39ef4ad3f6fb3fe17d29ef832 Nov 03 19:12:31.631031 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:31.630945 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" event={"ID":"313d392e-831e-428c-ad0d-0bbd03674212","Type":"ContainerStarted","Data":"6f7b3b53484d2caa4edbaa5911b59031a44359c39ef4ad3f6fb3fe17d29ef832"} Nov 03 19:12:31.631282 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:31.631257 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:31.632235 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:31.632218 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:12:32.640052 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:32.639964 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" event={"ID":"313d392e-831e-428c-ad0d-0bbd03674212","Type":"ContainerStarted","Data":"311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413"} Nov 03 19:12:32.640496 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:32.640050 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:32.783050 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:32.782986 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" podStartSLOduration=2.851365239 podStartE2EDuration="3.782971441s" totalImagesPullingTime="931.606202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:32.757418549 +0000 UTC m=+1686.729979729" watchObservedRunningTime="2026-11-03 19:12:32.782971441 +0000 UTC m=+1686.755532622" Nov 03 19:12:38.464626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:38.464585 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l"] Nov 03 19:12:38.465301 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:38.465267 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" containerID="cri-o://b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348" gracePeriod=30 Nov 03 19:12:39.026347 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:39.025386 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:12:41.300006 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:12:41.299930 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:12:43.783575 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.783551 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:12:43.928475 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.928445 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-token\" (UniqueName: \"kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-token\") pod \"ea2f83c9-6151-4788-a105-eb94e0e25754\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " Nov 03 19:12:43.928647 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.928529 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-kube-api-access-hrjxn\" (UniqueName: \"kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-kube-api-access-hrjxn\") pod \"ea2f83c9-6151-4788-a105-eb94e0e25754\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " Nov 03 19:12:43.928647 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.928581 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2f83c9-6151-4788-a105-eb94e0e25754-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea2f83c9-6151-4788-a105-eb94e0e25754-istiod-ca-cert\") pod \"ea2f83c9-6151-4788-a105-eb94e0e25754\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " Nov 03 19:12:43.928723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.928661 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-podinfo\") pod \"ea2f83c9-6151-4788-a105-eb94e0e25754\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " Nov 03 19:12:43.928723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.928702 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-data\") pod \"ea2f83c9-6151-4788-a105-eb94e0e25754\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " Nov 03 19:12:43.928797 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.928770 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-certs\") pod \"ea2f83c9-6151-4788-a105-eb94e0e25754\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " Nov 03 19:12:43.928861 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.928844 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-envoy\") pod \"ea2f83c9-6151-4788-a105-eb94e0e25754\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " Nov 03 19:12:43.928919 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.928889 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-socket\") pod \"ea2f83c9-6151-4788-a105-eb94e0e25754\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " Nov 03 19:12:43.928978 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.928924 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-credential-socket\") pod \"ea2f83c9-6151-4788-a105-eb94e0e25754\" (UID: \"ea2f83c9-6151-4788-a105-eb94e0e25754\") " Nov 03 19:12:43.929066 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.929017 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-data" pod "ea2f83c9-6151-4788-a105-eb94e0e25754" (UID: "ea2f83c9-6151-4788-a105-eb94e0e25754"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:43.929135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.929107 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-certs" pod "ea2f83c9-6151-4788-a105-eb94e0e25754" (UID: "ea2f83c9-6151-4788-a105-eb94e0e25754"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:43.929135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.929118 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2f83c9-6151-4788-a105-eb94e0e25754-istiod-ca-cert" pod "ea2f83c9-6151-4788-a105-eb94e0e25754" (UID: "ea2f83c9-6151-4788-a105-eb94e0e25754"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:12:43.929241 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.929228 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-socket" pod "ea2f83c9-6151-4788-a105-eb94e0e25754" (UID: "ea2f83c9-6151-4788-a105-eb94e0e25754"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:43.929438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.929411 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-credential-socket" pod "ea2f83c9-6151-4788-a105-eb94e0e25754" (UID: "ea2f83c9-6151-4788-a105-eb94e0e25754"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:43.929868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.929845 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea2f83c9-6151-4788-a105-eb94e0e25754-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:43.929967 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.929872 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:43.929967 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.929887 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:43.929967 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.929901 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:43.929967 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.929913 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:43.930946 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.930924 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-podinfo" pod "ea2f83c9-6151-4788-a105-eb94e0e25754" (UID: "ea2f83c9-6151-4788-a105-eb94e0e25754"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:12:43.931235 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.931215 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-token" pod "ea2f83c9-6151-4788-a105-eb94e0e25754" (UID: "ea2f83c9-6151-4788-a105-eb94e0e25754"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:43.931312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.931244 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-kube-api-access-hrjxn" pod "ea2f83c9-6151-4788-a105-eb94e0e25754" (UID: "ea2f83c9-6151-4788-a105-eb94e0e25754"). InnerVolumeSpecName "kube-api-access-hrjxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:12:43.931379 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:43.931318 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-envoy" pod "ea2f83c9-6151-4788-a105-eb94e0e25754" (UID: "ea2f83c9-6151-4788-a105-eb94e0e25754"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:12:44.031751 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.031660 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrjxn\" (UniqueName: \"kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-kube-api-access-hrjxn\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:44.031751 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.031695 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:44.031751 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.031709 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:44.031751 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.031723 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea2f83c9-6151-4788-a105-eb94e0e25754-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:12:44.391795 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.391700 5019 generic.go:356] "Generic (PLEG): container finished" podID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerID="b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348" exitCode=0 Nov 03 19:12:44.391795 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.391750 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" event={"ID":"ea2f83c9-6151-4788-a105-eb94e0e25754","Type":"ContainerDied","Data":"b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348"} Nov 03 19:12:44.391795 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.391756 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" Nov 03 19:12:44.392075 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.391805 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l" event={"ID":"ea2f83c9-6151-4788-a105-eb94e0e25754","Type":"ContainerDied","Data":"7f309e931af8b5af7a5c358de338bdba0e4d745a6aa42c8805a5a695730def76"} Nov 03 19:12:44.392075 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.391834 5019 scope.go:122] "RemoveContainer" containerID="b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348" Nov 03 19:12:44.402269 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.402243 5019 scope.go:122] "RemoveContainer" containerID="dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a" Nov 03 19:12:44.411373 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.411352 5019 scope.go:122] "RemoveContainer" containerID="b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348" Nov 03 19:12:44.411621 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:12:44.411602 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348\": container with ID starting with b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348 not found: ID does not exist" containerID="b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348" Nov 03 19:12:44.411674 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.411630 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348"} err="failed to get container status \"b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348\": rpc error: code = NotFound desc = could not find container \"b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348\": container with ID starting with b0f210b3142e48a627a9a6b21ea44694e361e37806d7a2297b80ba02f047e348 not found: ID does not exist" Nov 03 19:12:44.411674 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.411650 5019 scope.go:122] "RemoveContainer" containerID="dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a" Nov 03 19:12:44.411891 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:12:44.411872 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a\": container with ID starting with dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a not found: ID does not exist" containerID="dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a" Nov 03 19:12:44.411944 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.411901 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a"} err="failed to get container status \"dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a\": rpc error: code = NotFound desc = could not find container \"dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a\": container with ID starting with dbf76e2b6a6fe8ba25da632d257c45aeffd9351443830c802a353f020fb9411a not found: ID does not exist" Nov 03 19:12:44.550512 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.550477 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l"] Nov 03 19:12:44.598977 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:44.598946 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7a8bab4a-openshift-default-764575d85c-q7r9l"] Nov 03 19:12:46.152218 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:46.152186 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" path="/var/lib/kubelet/pods/ea2f83c9-6151-4788-a105-eb94e0e25754/volumes" Nov 03 19:12:53.952263 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:53.952226 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg"] Nov 03 19:12:53.953340 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:53.953299 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" Nov 03 19:12:53.953340 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:53.953340 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" Nov 03 19:12:53.953542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:53.953408 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" Nov 03 19:12:53.953542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:53.953417 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" Nov 03 19:12:53.953642 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:53.953630 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" Nov 03 19:12:53.953692 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:53.953657 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2f83c9-6151-4788-a105-eb94e0e25754" containerName="istio-proxy" Nov 03 19:12:53.957818 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:53.957793 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.006600 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.006564 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-credential-socket\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.006772 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.006660 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f927r\" (UniqueName: \"kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-kube-api-access-f927r\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.006772 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.006752 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-socket\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.006879 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.006820 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-envoy\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.006993 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.006913 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-certs\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.006993 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.006970 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-data\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.007149 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.007019 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a00172-f381-4936-8800-18bdd9f428d6-istio-podinfo\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.007149 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.007073 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a00172-f381-4936-8800-18bdd9f428d6-istiod-ca-cert\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.007149 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.007100 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-istio-token\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.017496 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.017465 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4702dd-openshift-default-dockercfg-k8gk7\"" Nov 03 19:12:54.083520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.083483 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg"] Nov 03 19:12:54.124355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.118605 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a00172-f381-4936-8800-18bdd9f428d6-istio-podinfo\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.124355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.118690 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a00172-f381-4936-8800-18bdd9f428d6-istiod-ca-cert\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.124355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.118721 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-istio-token\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.124355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.118903 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-credential-socket\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.124355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.118990 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f927r\" (UniqueName: \"kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-kube-api-access-f927r\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.124355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.119081 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-socket\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.124355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.119155 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-envoy\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.124355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.119259 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-certs\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.124355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.119316 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-data\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.124355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.119876 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-data\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.125009 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.124516 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a00172-f381-4936-8800-18bdd9f428d6-istiod-ca-cert\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.125073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.125014 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-credential-socket\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.144834 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.144796 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-socket\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.216678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.216595 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-istio-token\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.218555 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.218521 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-certs\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.341350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.317290 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f927r\" (UniqueName: \"kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-kube-api-access-f927r\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.399048 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.398961 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-envoy\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.407138 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.407106 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a00172-f381-4936-8800-18bdd9f428d6-istio-podinfo\") pod \"e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:54.613415 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:54.612615 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:55.103764 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:12:55.103715 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a00172_f381_4936_8800_18bdd9f428d6.slice/crio-2e91302933abba8e8101be331b798d1578bbdfffacbed60142f21dda151a0af2 WatchSource:0}: Error finding container 2e91302933abba8e8101be331b798d1578bbdfffacbed60142f21dda151a0af2: Status 404 returned error can't find the container with id 2e91302933abba8e8101be331b798d1578bbdfffacbed60142f21dda151a0af2 Nov 03 19:12:55.104212 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:55.104031 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg"] Nov 03 19:12:55.107119 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:55.107073 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:12:55.107219 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:55.107169 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:12:55.107219 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:55.107213 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:12:55.548419 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:55.546553 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" event={"ID":"b8a00172-f381-4936-8800-18bdd9f428d6","Type":"ContainerStarted","Data":"6dbdeb05a98c74bcf6a1bf259446849ac996f006ca2d585294941a6e88a57709"} Nov 03 19:12:55.548419 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:55.546603 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" event={"ID":"b8a00172-f381-4936-8800-18bdd9f428d6","Type":"ContainerStarted","Data":"2e91302933abba8e8101be331b798d1578bbdfffacbed60142f21dda151a0af2"} Nov 03 19:12:55.613375 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:55.613341 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:55.754927 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:55.754874 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" podStartSLOduration=2.754860677 podStartE2EDuration="2.754860677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:55.706199911 +0000 UTC m=+1709.678761093" watchObservedRunningTime="2026-11-03 19:12:55.754860677 +0000 UTC m=+1709.727421858" Nov 03 19:12:56.619214 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:56.619181 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:57.563392 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:57.563363 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:12:57.564451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:12:57.564426 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:13:00.041126 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.041090 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz"] Nov 03 19:13:00.046561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.046533 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.102893 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.102863 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5b012841-serving-cert\"" Nov 03 19:13:00.135495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.135457 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-openshift-service-ca\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.135696 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.135532 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-maas-api-tls\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.135806 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.135773 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f79pg\" (UniqueName: \"kubernetes.io/projected/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-kube-api-access-f79pg\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.136019 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.135997 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-metrics-certs\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.170726 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.170692 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz"] Nov 03 19:13:00.237612 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.237578 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-openshift-service-ca\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.237806 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.237638 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-maas-api-tls\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.237806 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.237766 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f79pg\" (UniqueName: \"kubernetes.io/projected/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-kube-api-access-f79pg\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.237995 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.237975 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-metrics-certs\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.238140 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:13:00.238124 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-serving-cert: secret "maas-api-e2e-derive-5b012841-serving-cert" not found Nov 03 19:13:00.238304 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:13:00.238290 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-maas-api-tls podName:65005b3c-db12-4ddc-ab6e-2a4261ee6b69 nodeName:}" failed. No retries permitted until 2026-11-03 19:13:00.738264045 +0000 UTC m=+1714.710825213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-maas-api-tls") pod "maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" (UID: "65005b3c-db12-4ddc-ab6e-2a4261ee6b69") : secret "maas-api-e2e-derive-5b012841-serving-cert" not found Nov 03 19:13:00.239475 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.239450 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-openshift-service-ca\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.254050 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.254026 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-metrics-certs\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.399409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.382492 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f79pg\" (UniqueName: \"kubernetes.io/projected/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-kube-api-access-f79pg\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.752656 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.752616 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl"] Nov 03 19:13:00.752923 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.752851 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" podUID="edcef2b9-e070-4b5b-9902-323feda55bdf" containerName="istio-proxy" containerID="cri-o://2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638" gracePeriod=30 Nov 03 19:13:00.757535 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.757511 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q"] Nov 03 19:13:00.757719 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.757699 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" podUID="a160d419-2de3-4dc1-affb-e50a0867becb" containerName="istio-proxy" containerID="cri-o://76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437" gracePeriod=30 Nov 03 19:13:00.772596 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.772565 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-maas-api-tls\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.775120 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.775091 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-maas-api-tls\") pod \"maas-api-e2e-derive-5b012841-7c4f66646-vbcnz\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:00.962581 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:00.962547 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:01.343645 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:13:01.342887 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65005b3c_db12_4ddc_ab6e_2a4261ee6b69.slice/crio-08b9fcd5a21be03ea4a9790aee17d96c2cbfcd0bf545354e2d420be91af7b335 WatchSource:0}: Error finding container 08b9fcd5a21be03ea4a9790aee17d96c2cbfcd0bf545354e2d420be91af7b335: Status 404 returned error can't find the container with id 08b9fcd5a21be03ea4a9790aee17d96c2cbfcd0bf545354e2d420be91af7b335 Nov 03 19:13:01.343645 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:01.343075 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz"] Nov 03 19:13:01.631259 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:01.631169 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" event={"ID":"65005b3c-db12-4ddc-ab6e-2a4261ee6b69","Type":"ContainerStarted","Data":"08b9fcd5a21be03ea4a9790aee17d96c2cbfcd0bf545354e2d420be91af7b335"} Nov 03 19:13:02.639854 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:02.639819 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" event={"ID":"65005b3c-db12-4ddc-ab6e-2a4261ee6b69","Type":"ContainerStarted","Data":"047e009872e0ada895ce266833d3524ebc5399a914ff3571d7701d915d15a2f0"} Nov 03 19:13:02.640366 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:02.639879 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:02.754918 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:02.754864 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" podStartSLOduration=2.937268456 podStartE2EDuration="3.754849082s" totalImagesPullingTime="817.580626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:02.750485971 +0000 UTC m=+1716.723047153" watchObservedRunningTime="2026-11-03 19:13:02.754849082 +0000 UTC m=+1716.727410263" Nov 03 19:13:05.658348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:05.652829 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:13:05.658348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:05.652903 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" podUID="a160d419-2de3-4dc1-affb-e50a0867becb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:13:06.294554 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.294525 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:13:06.298556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.298535 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:13:06.309112 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309089 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-podinfo\") pod \"edcef2b9-e070-4b5b-9902-323feda55bdf\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " Nov 03 19:13:06.309245 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309130 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-certs\") pod \"a160d419-2de3-4dc1-affb-e50a0867becb\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " Nov 03 19:13:06.309245 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309157 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-envoy\") pod \"a160d419-2de3-4dc1-affb-e50a0867becb\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " Nov 03 19:13:06.309418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309249 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-certs\") pod \"edcef2b9-e070-4b5b-9902-323feda55bdf\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " Nov 03 19:13:06.309418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309286 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-credential-socket\") pod \"a160d419-2de3-4dc1-affb-e50a0867becb\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " Nov 03 19:13:06.309418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309315 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-socket\") pod \"edcef2b9-e070-4b5b-9902-323feda55bdf\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " Nov 03 19:13:06.309579 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309429 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-envoy\") pod \"edcef2b9-e070-4b5b-9902-323feda55bdf\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " Nov 03 19:13:06.309579 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309479 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-data\") pod \"edcef2b9-e070-4b5b-9902-323feda55bdf\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " Nov 03 19:13:06.309579 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309559 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-istio-token\" (UniqueName: \"kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-istio-token\") pod \"a160d419-2de3-4dc1-affb-e50a0867becb\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " Nov 03 19:13:06.309724 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309633 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-credential-socket\") pod \"edcef2b9-e070-4b5b-9902-323feda55bdf\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " Nov 03 19:13:06.309724 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309683 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a160d419-2de3-4dc1-affb-e50a0867becb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a160d419-2de3-4dc1-affb-e50a0867becb-istio-podinfo\") pod \"a160d419-2de3-4dc1-affb-e50a0867becb\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " Nov 03 19:13:06.309821 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309759 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-data\") pod \"a160d419-2de3-4dc1-affb-e50a0867becb\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " Nov 03 19:13:06.309821 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309804 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-kube-api-access-qrtxb\" (UniqueName: \"kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-kube-api-access-qrtxb\") pod \"a160d419-2de3-4dc1-affb-e50a0867becb\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " Nov 03 19:13:06.309925 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309853 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-kube-api-access-zklh4\" (UniqueName: \"kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-kube-api-access-zklh4\") pod \"edcef2b9-e070-4b5b-9902-323feda55bdf\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " Nov 03 19:13:06.309925 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309887 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edcef2b9-e070-4b5b-9902-323feda55bdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcef2b9-e070-4b5b-9902-323feda55bdf-istiod-ca-cert\") pod \"edcef2b9-e070-4b5b-9902-323feda55bdf\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " Nov 03 19:13:06.310022 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309995 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a160d419-2de3-4dc1-affb-e50a0867becb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a160d419-2de3-4dc1-affb-e50a0867becb-istiod-ca-cert\") pod \"a160d419-2de3-4dc1-affb-e50a0867becb\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " Nov 03 19:13:06.310080 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.310037 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-token\" (UniqueName: \"kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-token\") pod \"edcef2b9-e070-4b5b-9902-323feda55bdf\" (UID: \"edcef2b9-e070-4b5b-9902-323feda55bdf\") " Nov 03 19:13:06.310135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.310124 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-socket\") pod \"a160d419-2de3-4dc1-affb-e50a0867becb\" (UID: \"a160d419-2de3-4dc1-affb-e50a0867becb\") " Nov 03 19:13:06.310187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309474 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-certs" pod "a160d419-2de3-4dc1-affb-e50a0867becb" (UID: "a160d419-2de3-4dc1-affb-e50a0867becb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:06.310237 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309697 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-credential-socket" pod "a160d419-2de3-4dc1-affb-e50a0867becb" (UID: "a160d419-2de3-4dc1-affb-e50a0867becb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:06.310237 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.309855 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-certs" pod "edcef2b9-e070-4b5b-9902-323feda55bdf" (UID: "edcef2b9-e070-4b5b-9902-323feda55bdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:06.310237 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.310009 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-credential-socket" pod "edcef2b9-e070-4b5b-9902-323feda55bdf" (UID: "edcef2b9-e070-4b5b-9902-323feda55bdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:06.310760 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.310733 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edcef2b9-e070-4b5b-9902-323feda55bdf-istiod-ca-cert" pod "edcef2b9-e070-4b5b-9902-323feda55bdf" (UID: "edcef2b9-e070-4b5b-9902-323feda55bdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:06.311157 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.311132 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.311248 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.311160 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcef2b9-e070-4b5b-9902-323feda55bdf-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.311248 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.311176 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.311248 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.311190 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.311248 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.311204 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.312048 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.311853 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-podinfo" pod "edcef2b9-e070-4b5b-9902-323feda55bdf" (UID: "edcef2b9-e070-4b5b-9902-323feda55bdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:13:06.312048 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.310163 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-data" pod "edcef2b9-e070-4b5b-9902-323feda55bdf" (UID: "edcef2b9-e070-4b5b-9902-323feda55bdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:06.312191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.312105 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-envoy" pod "edcef2b9-e070-4b5b-9902-323feda55bdf" (UID: "edcef2b9-e070-4b5b-9902-323feda55bdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:06.312191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.312126 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-socket" pod "edcef2b9-e070-4b5b-9902-323feda55bdf" (UID: "edcef2b9-e070-4b5b-9902-323feda55bdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:06.312298 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.312185 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-socket" pod "a160d419-2de3-4dc1-affb-e50a0867becb" (UID: "a160d419-2de3-4dc1-affb-e50a0867becb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:06.312298 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.312226 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-data" pod "a160d419-2de3-4dc1-affb-e50a0867becb" (UID: "a160d419-2de3-4dc1-affb-e50a0867becb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:06.312593 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.312566 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a160d419-2de3-4dc1-affb-e50a0867becb-istio-podinfo" pod "a160d419-2de3-4dc1-affb-e50a0867becb" (UID: "a160d419-2de3-4dc1-affb-e50a0867becb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:13:06.312593 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.312581 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a160d419-2de3-4dc1-affb-e50a0867becb-istiod-ca-cert" pod "a160d419-2de3-4dc1-affb-e50a0867becb" (UID: "a160d419-2de3-4dc1-affb-e50a0867becb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:06.312904 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.312881 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-envoy" pod "a160d419-2de3-4dc1-affb-e50a0867becb" (UID: "a160d419-2de3-4dc1-affb-e50a0867becb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:06.313652 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.313625 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-kube-api-access-zklh4" pod "edcef2b9-e070-4b5b-9902-323feda55bdf" (UID: "edcef2b9-e070-4b5b-9902-323feda55bdf"). InnerVolumeSpecName "kube-api-access-zklh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:06.313733 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.313649 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-token" pod "edcef2b9-e070-4b5b-9902-323feda55bdf" (UID: "edcef2b9-e070-4b5b-9902-323feda55bdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:06.313825 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.313807 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-kube-api-access-qrtxb" pod "a160d419-2de3-4dc1-affb-e50a0867becb" (UID: "a160d419-2de3-4dc1-affb-e50a0867becb"). InnerVolumeSpecName "kube-api-access-qrtxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:06.314809 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.314779 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-istio-token" pod "a160d419-2de3-4dc1-affb-e50a0867becb" (UID: "a160d419-2de3-4dc1-affb-e50a0867becb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:06.413141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413100 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.413141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413135 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.413396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413150 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.413396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413163 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.413396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413174 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a160d419-2de3-4dc1-affb-e50a0867becb-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.413396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413187 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.413396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413199 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrtxb\" (UniqueName: \"kubernetes.io/projected/a160d419-2de3-4dc1-affb-e50a0867becb-kube-api-access-qrtxb\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.413396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413213 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zklh4\" (UniqueName: \"kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-kube-api-access-zklh4\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.413396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413225 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a160d419-2de3-4dc1-affb-e50a0867becb-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.413396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413237 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.413396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413248 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.413396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413261 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcef2b9-e070-4b5b-9902-323feda55bdf-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.413396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.413273 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a160d419-2de3-4dc1-affb-e50a0867becb-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:06.796317 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.796282 5019 generic.go:356] "Generic (PLEG): container finished" podID="edcef2b9-e070-4b5b-9902-323feda55bdf" containerID="2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638" exitCode=0 Nov 03 19:13:06.796778 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.796340 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" Nov 03 19:13:06.796778 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.796359 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" event={"ID":"edcef2b9-e070-4b5b-9902-323feda55bdf","Type":"ContainerDied","Data":"2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638"} Nov 03 19:13:06.796778 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.796422 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl" event={"ID":"edcef2b9-e070-4b5b-9902-323feda55bdf","Type":"ContainerDied","Data":"359933c19d47f24899fd395d19923abc2575c022395bb5f44992c6fa4df54405"} Nov 03 19:13:06.796778 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.796503 5019 scope.go:122] "RemoveContainer" containerID="2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638" Nov 03 19:13:06.800094 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.800072 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" Nov 03 19:13:06.800256 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.800230 5019 generic.go:356] "Generic (PLEG): container finished" podID="a160d419-2de3-4dc1-affb-e50a0867becb" containerID="76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437" exitCode=0 Nov 03 19:13:06.800369 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.800280 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" event={"ID":"a160d419-2de3-4dc1-affb-e50a0867becb","Type":"ContainerDied","Data":"76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437"} Nov 03 19:13:06.800369 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.800337 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q" event={"ID":"a160d419-2de3-4dc1-affb-e50a0867becb","Type":"ContainerDied","Data":"3bc9819ba2bd7d4e082e00b7f79f34f4666124158e75eb471f093f4514170361"} Nov 03 19:13:06.816059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.816038 5019 scope.go:122] "RemoveContainer" containerID="2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638" Nov 03 19:13:06.816450 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:13:06.816422 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638\": container with ID starting with 2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638 not found: ID does not exist" containerID="2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638" Nov 03 19:13:06.816552 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.816458 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638"} err="failed to get container status \"2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638\": rpc error: code = NotFound desc = could not find container \"2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638\": container with ID starting with 2478a84e71d1456db03e22ee2b4678f653fa3eb9a8f2c49996d54c4212ede638 not found: ID does not exist" Nov 03 19:13:06.816552 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.816482 5019 scope.go:122] "RemoveContainer" containerID="76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437" Nov 03 19:13:06.827190 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.827174 5019 scope.go:122] "RemoveContainer" containerID="76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437" Nov 03 19:13:06.827441 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:13:06.827422 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437\": container with ID starting with 76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437 not found: ID does not exist" containerID="76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437" Nov 03 19:13:06.827486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.827447 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437"} err="failed to get container status \"76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437\": rpc error: code = NotFound desc = could not find container \"76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437\": container with ID starting with 76f442329ce156a9f9d16dda989af7aaf20ee25e7183d4406cb70f0c8649b437 not found: ID does not exist" Nov 03 19:13:06.954017 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.953985 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl"] Nov 03 19:13:06.958142 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:06.958114 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-6kwtl"] Nov 03 19:13:07.062317 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:07.062208 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q"] Nov 03 19:13:07.066783 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:07.066756 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-pbl6q"] Nov 03 19:13:08.264296 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:08.263441 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a160d419-2de3-4dc1-affb-e50a0867becb" path="/var/lib/kubelet/pods/a160d419-2de3-4dc1-affb-e50a0867becb/volumes" Nov 03 19:13:08.283293 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:08.264317 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edcef2b9-e070-4b5b-9902-323feda55bdf" path="/var/lib/kubelet/pods/edcef2b9-e070-4b5b-9902-323feda55bdf/volumes" Nov 03 19:13:08.654189 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:08.654120 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:13:11.409314 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:13:11.409270 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:13:17.599526 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.599489 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982"] Nov 03 19:13:17.600297 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.600279 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a160d419-2de3-4dc1-affb-e50a0867becb" containerName="istio-proxy" Nov 03 19:13:17.600297 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.600297 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a160d419-2de3-4dc1-affb-e50a0867becb" containerName="istio-proxy" Nov 03 19:13:17.600521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.600319 5019 cpu_manager.go:414] "containerMap: removing container" podUID="edcef2b9-e070-4b5b-9902-323feda55bdf" containerName="istio-proxy" Nov 03 19:13:17.600521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.600342 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edcef2b9-e070-4b5b-9902-323feda55bdf" containerName="istio-proxy" Nov 03 19:13:17.600634 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.600527 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a160d419-2de3-4dc1-affb-e50a0867becb" containerName="istio-proxy" Nov 03 19:13:17.600634 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.600543 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="edcef2b9-e070-4b5b-9902-323feda55bdf" containerName="istio-proxy" Nov 03 19:13:17.604441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.604421 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.659159 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.659124 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-08baeb-openshift-default-dockercfg-rtbsd\"" Nov 03 19:13:17.709401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.709357 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982"] Nov 03 19:13:17.854147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.852596 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istiod-ca-cert\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.854147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.852723 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-credential-socket\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.854147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.852754 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-socket\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.854147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.852950 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-envoy\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.854147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.853003 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-token\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.854147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.853119 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-data\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.854147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.853158 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-certs\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.854147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.853308 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc7sl\" (UniqueName: \"kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-kube-api-access-dc7sl\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.854147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.853364 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-podinfo\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.955475 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.955440 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-envoy\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.955728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.955709 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-token\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.955921 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.955900 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-data\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.956078 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.956059 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-certs\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.956312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.956294 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc7sl\" (UniqueName: \"kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-kube-api-access-dc7sl\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.956508 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.956480 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-podinfo\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.956728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.956704 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istiod-ca-cert\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:17.956940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.956890 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-data\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:18.046487 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.956903 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-credential-socket\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:18.046801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:18.046762 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-socket\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:18.047181 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.957540 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-certs\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:18.047298 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.971011 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istiod-ca-cert\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:18.047417 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:17.957185 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-credential-socket\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:18.047881 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:18.047852 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-socket\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:18.118150 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:18.118058 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-envoy\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:18.120175 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:18.120148 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-podinfo\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:18.128807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:18.128775 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc7sl\" (UniqueName: \"kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-kube-api-access-dc7sl\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:18.130929 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:18.130906 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-token\") pod \"e2e-api-a-08baeb-openshift-default-748568f9f7-jt982\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:18.221135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:18.221102 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:18.666928 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:13:18.666891 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cdf3bc_673f_4fd1_bb49_92b3e4994229.slice/crio-fdf2724493cf8d39073c32d5115d94e5c9a17a105e801152cb647454dd6995ab WatchSource:0}: Error finding container fdf2724493cf8d39073c32d5115d94e5c9a17a105e801152cb647454dd6995ab: Status 404 returned error can't find the container with id fdf2724493cf8d39073c32d5115d94e5c9a17a105e801152cb647454dd6995ab Nov 03 19:13:18.668680 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:18.668651 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982"] Nov 03 19:13:18.670031 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:18.669994 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:13:18.670118 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:18.670072 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:13:18.670193 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:18.670115 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:13:19.139794 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:19.139759 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" event={"ID":"c0cdf3bc-673f-4fd1-bb49-92b3e4994229","Type":"ContainerStarted","Data":"b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1"} Nov 03 19:13:19.139951 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:19.139823 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" event={"ID":"c0cdf3bc-673f-4fd1-bb49-92b3e4994229","Type":"ContainerStarted","Data":"fdf2724493cf8d39073c32d5115d94e5c9a17a105e801152cb647454dd6995ab"} Nov 03 19:13:19.221477 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:19.221444 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:19.227231 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:19.227205 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:19.316862 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:19.316808 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" podStartSLOduration=2.316795227 podStartE2EDuration="2.316795227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:19.312526125 +0000 UTC m=+1733.285087307" watchObservedRunningTime="2026-11-03 19:13:19.316795227 +0000 UTC m=+1733.289356412" Nov 03 19:13:20.152990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:20.152962 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:20.153384 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:20.153033 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:13:26.214976 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.214941 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6"] Nov 03 19:13:26.225059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.225012 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6"] Nov 03 19:13:26.256720 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.225816 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:26.318730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.318699 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d86d1cd0-serving-cert\"" Nov 03 19:13:26.468824 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.343712 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-openshift-service-ca\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:26.469103 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.469076 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-metrics-certs\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:26.470780 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.469584 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2rbm\" (UniqueName: \"kubernetes.io/projected/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-kube-api-access-g2rbm\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:26.494647 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.494611 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-maas-api-tls\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:26.605460 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.601769 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2rbm\" (UniqueName: \"kubernetes.io/projected/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-kube-api-access-g2rbm\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:26.605460 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.601918 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-maas-api-tls\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:26.605460 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.602154 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-openshift-service-ca\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:26.605460 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.602192 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-metrics-certs\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:26.605460 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:13:26.602947 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-serving-cert: secret "maas-api-e2e-mt-d86d1cd0-serving-cert" not found Nov 03 19:13:26.605460 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:13:26.603034 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-maas-api-tls podName:1d55ab25-8642-4b74-99c1-9aaf2eed7a15 nodeName:}" failed. No retries permitted until 2026-11-03 19:13:27.103009199 +0000 UTC m=+1741.075570364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-maas-api-tls") pod "maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" (UID: "1d55ab25-8642-4b74-99c1-9aaf2eed7a15") : secret "maas-api-e2e-mt-d86d1cd0-serving-cert" not found Nov 03 19:13:26.605460 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.604244 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-openshift-service-ca\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:26.662439 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.640686 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-metrics-certs\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:26.686651 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:26.686610 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2rbm\" (UniqueName: \"kubernetes.io/projected/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-kube-api-access-g2rbm\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:27.129666 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:27.129633 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-maas-api-tls\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:27.132047 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:27.132023 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-maas-api-tls\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:27.348562 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:27.348526 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:27.777734 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:27.777701 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6"] Nov 03 19:13:27.780654 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:13:27.780624 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d55ab25_8642_4b74_99c1_9aaf2eed7a15.slice/crio-221bac659712ee5811e418cc52ac42ab6f541d3d7f8e52dc76ca08048ebe3fb7 WatchSource:0}: Error finding container 221bac659712ee5811e418cc52ac42ab6f541d3d7f8e52dc76ca08048ebe3fb7: Status 404 returned error can't find the container with id 221bac659712ee5811e418cc52ac42ab6f541d3d7f8e52dc76ca08048ebe3fb7 Nov 03 19:13:28.472866 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:28.472825 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" event={"ID":"1d55ab25-8642-4b74-99c1-9aaf2eed7a15","Type":"ContainerStarted","Data":"221bac659712ee5811e418cc52ac42ab6f541d3d7f8e52dc76ca08048ebe3fb7"} Nov 03 19:13:29.482049 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:29.482007 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" event={"ID":"1d55ab25-8642-4b74-99c1-9aaf2eed7a15","Type":"ContainerStarted","Data":"eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5"} Nov 03 19:13:29.482573 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:29.482198 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:29.594301 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:29.594191 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" podStartSLOduration=2.154636952 podStartE2EDuration="3.594177919s" totalImagesPullingTime="1.439540967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:29.591817143 +0000 UTC m=+1743.564378325" watchObservedRunningTime="2026-11-03 19:13:29.594177919 +0000 UTC m=+1743.566739101" Nov 03 19:13:35.494706 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:35.494670 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:13:41.808409 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:13:41.804522 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:13:56.139730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.139693 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf"] Nov 03 19:13:56.147742 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.147718 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.210740 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.210710 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-4702dd-serving-cert\"" Nov 03 19:13:56.251750 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.251714 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf"] Nov 03 19:13:56.338057 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.338021 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n25hh\" (UniqueName: \"kubernetes.io/projected/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-kube-api-access-n25hh\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.338231 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.338175 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.338374 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.338272 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.338608 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.338584 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.472361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.449424 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.472685 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:13:56.451176 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-serving-cert: secret "maas-api-e2e-shared-a-w2-4702dd-serving-cert" not found Nov 03 19:13:56.472911 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:13:56.472895 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-maas-api-tls podName:dffda2a7-8f0f-42a5-b6df-5b656f40ee22 nodeName:}" failed. No retries permitted until 2026-11-03 19:13:56.972866127 +0000 UTC m=+1770.945427307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-maas-api-tls") pod "maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" (UID: "dffda2a7-8f0f-42a5-b6df-5b656f40ee22") : secret "maas-api-e2e-shared-a-w2-4702dd-serving-cert" not found Nov 03 19:13:56.476442 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.476413 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.476893 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.476860 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.477179 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.477150 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n25hh\" (UniqueName: \"kubernetes.io/projected/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-kube-api-access-n25hh\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.496352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.493610 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.536579 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.531946 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.549754 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.549046 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n25hh\" (UniqueName: \"kubernetes.io/projected/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-kube-api-access-n25hh\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.997084 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.997033 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:56.999520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:56.999490 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:57.062999 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:57.062969 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:57.442626 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:13:57.442588 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffda2a7_8f0f_42a5_b6df_5b656f40ee22.slice/crio-fcde4d550e4eb7a9bc406ccc7a84fd6902f62e1a73175aadac52a430b2bdc15f WatchSource:0}: Error finding container fcde4d550e4eb7a9bc406ccc7a84fd6902f62e1a73175aadac52a430b2bdc15f: Status 404 returned error can't find the container with id fcde4d550e4eb7a9bc406ccc7a84fd6902f62e1a73175aadac52a430b2bdc15f Nov 03 19:13:57.443501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:57.443412 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf"] Nov 03 19:13:57.911621 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:57.911533 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" event={"ID":"dffda2a7-8f0f-42a5-b6df-5b656f40ee22","Type":"ContainerStarted","Data":"fcde4d550e4eb7a9bc406ccc7a84fd6902f62e1a73175aadac52a430b2bdc15f"} Nov 03 19:13:58.920533 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:58.920493 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" event={"ID":"dffda2a7-8f0f-42a5-b6df-5b656f40ee22","Type":"ContainerStarted","Data":"bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881"} Nov 03 19:13:58.920936 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:58.920595 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:13:59.033650 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:13:59.033597 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" podStartSLOduration=2.101595583 podStartE2EDuration="3.033583161s" totalImagesPullingTime="931.987578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:59.031281472 +0000 UTC m=+1773.003842653" watchObservedRunningTime="2026-11-03 19:13:59.033583161 +0000 UTC m=+1773.006144341" Nov 03 19:14:01.946415 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:01.946383 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb"] Nov 03 19:14:01.950742 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:01.950720 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.015627 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.015559 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bbccf3-openshift-default-dockercfg-g6xwg\"" Nov 03 19:14:02.027595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.027564 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-token\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.027595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.027597 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-podinfo\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.027833 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.027640 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-envoy\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.027920 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.027892 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-certs\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.027982 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.027949 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rztvl\" (UniqueName: \"kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-kube-api-access-rztvl\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.028085 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.028067 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-data\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.028145 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.028110 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-socket\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.028226 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.028211 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istiod-ca-cert\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.028283 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.028269 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-credential-socket\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.061777 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.061740 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb"] Nov 03 19:14:02.130672 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.130628 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-certs\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.130853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.130716 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rztvl\" (UniqueName: \"kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-kube-api-access-rztvl\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.130853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.130801 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-data\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.130958 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.130857 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-socket\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.130958 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.130925 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istiod-ca-cert\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.131060 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.130982 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-credential-socket\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.131120 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.131100 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-certs\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.131194 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.131173 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-token\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.131256 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.131208 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-podinfo\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.131308 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.131260 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-envoy\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.131856 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.131473 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-socket\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.131856 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.131813 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-credential-socket\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.132019 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.131875 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-data\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.132075 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.132013 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istiod-ca-cert\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.136205 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.136182 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-podinfo\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.139795 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.139774 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-envoy\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.345481 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.321892 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rztvl\" (UniqueName: \"kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-kube-api-access-rztvl\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.345481 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.324714 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-token\") pod \"e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.345481 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.327262 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:02.845486 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:14:02.845455 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9a1357_9f0e_4616_8e9a_6c60f4385cec.slice/crio-aa85916419b876d9301dae69ee7b635999394cde13e7b9f14ab55e6b9c5f2c28 WatchSource:0}: Error finding container aa85916419b876d9301dae69ee7b635999394cde13e7b9f14ab55e6b9c5f2c28: Status 404 returned error can't find the container with id aa85916419b876d9301dae69ee7b635999394cde13e7b9f14ab55e6b9c5f2c28 Nov 03 19:14:02.845726 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.845701 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb"] Nov 03 19:14:02.848361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.848293 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:14:02.848445 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.848373 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:14:02.848445 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.848406 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:14:02.957993 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.957953 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" event={"ID":"dd9a1357-9f0e-4616-8e9a-6c60f4385cec","Type":"ContainerStarted","Data":"3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e"} Nov 03 19:14:02.958439 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:02.958062 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" event={"ID":"dd9a1357-9f0e-4616-8e9a-6c60f4385cec","Type":"ContainerStarted","Data":"aa85916419b876d9301dae69ee7b635999394cde13e7b9f14ab55e6b9c5f2c28"} Nov 03 19:14:03.083563 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:03.083503 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" podStartSLOduration=2.083483063 podStartE2EDuration="2.083483063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:14:03.077691965 +0000 UTC m=+1777.050253151" watchObservedRunningTime="2026-11-03 19:14:03.083483063 +0000 UTC m=+1777.056044246" Nov 03 19:14:03.350272 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:03.343706 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:03.355962 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:03.355929 5019 patch_prober.go:29] interesting pod/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:14:03.356165 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:03.355988 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:14:04.344417 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:04.344386 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:04.934222 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:04.934191 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:14:04.974612 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:04.974574 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:04.975783 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:04.975762 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:14:11.298268 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:14:11.298225 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:14:16.374638 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:16.374599 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52"] Nov 03 19:14:16.375083 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:16.374861 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" podUID="44eda232-a5c8-4016-866b-21c73206765a" containerName="istio-proxy" containerID="cri-o://87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998" gracePeriod=30 Nov 03 19:14:16.418648 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:16.418614 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp"] Nov 03 19:14:16.418847 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:16.418826 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" podUID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" containerName="istio-proxy" containerID="cri-o://dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae" gracePeriod=30 Nov 03 19:14:16.425092 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:16.425058 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf"] Nov 03 19:14:16.425367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:16.425278 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" podUID="7a2a7348-14ea-449a-9988-99ff5f48506f" containerName="istio-proxy" containerID="cri-o://57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9" gracePeriod=30 Nov 03 19:14:21.507687 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:21.507651 5019 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:14:21.508152 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:21.507721 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" podUID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:14:22.153721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.153682 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:14:22.170752 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.170712 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:14:22.219553 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.219518 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:14:22.287685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.287600 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-socket\") pod \"44eda232-a5c8-4016-866b-21c73206765a\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " Nov 03 19:14:22.287685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.287644 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-socket\") pod \"7a2a7348-14ea-449a-9988-99ff5f48506f\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " Nov 03 19:14:22.287924 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.287706 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-socket\") pod \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " Nov 03 19:14:22.287924 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.287735 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-podinfo\") pod \"7a2a7348-14ea-449a-9988-99ff5f48506f\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " Nov 03 19:14:22.287924 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.287776 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-kube-api-access-z72xp\" (UniqueName: \"kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-kube-api-access-z72xp\") pod \"44eda232-a5c8-4016-866b-21c73206765a\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " Nov 03 19:14:22.287924 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.287810 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44eda232-a5c8-4016-866b-21c73206765a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eda232-a5c8-4016-866b-21c73206765a-istiod-ca-cert\") pod \"44eda232-a5c8-4016-866b-21c73206765a\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " Nov 03 19:14:22.287924 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.287861 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istiod-ca-cert\") pod \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " Nov 03 19:14:22.287924 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.287863 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-socket" pod "44eda232-a5c8-4016-866b-21c73206765a" (UID: "44eda232-a5c8-4016-866b-21c73206765a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.287924 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.287891 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2a7348-14ea-449a-9988-99ff5f48506f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2a7348-14ea-449a-9988-99ff5f48506f-istiod-ca-cert\") pod \"7a2a7348-14ea-449a-9988-99ff5f48506f\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " Nov 03 19:14:22.288284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.287942 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-certs\") pod \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " Nov 03 19:14:22.288284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288003 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-envoy\") pod \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " Nov 03 19:14:22.288284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288057 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-podinfo\") pod \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " Nov 03 19:14:22.288284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288115 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-credential-socket\") pod \"44eda232-a5c8-4016-866b-21c73206765a\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " Nov 03 19:14:22.288284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288174 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-envoy\") pod \"44eda232-a5c8-4016-866b-21c73206765a\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " Nov 03 19:14:22.288284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288210 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-data\") pod \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " Nov 03 19:14:22.288284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288275 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-istio-token\" (UniqueName: \"kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-istio-token\") pod \"44eda232-a5c8-4016-866b-21c73206765a\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288002 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-socket" pod "7a2a7348-14ea-449a-9988-99ff5f48506f" (UID: "7a2a7348-14ea-449a-9988-99ff5f48506f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288286 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istiod-ca-cert" pod "5c37240b-0a43-4d38-a54e-47fdb0ff5c69" (UID: "5c37240b-0a43-4d38-a54e-47fdb0ff5c69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288314 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-certs\") pod \"44eda232-a5c8-4016-866b-21c73206765a\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288348 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44eda232-a5c8-4016-866b-21c73206765a-istiod-ca-cert" pod "44eda232-a5c8-4016-866b-21c73206765a" (UID: "44eda232-a5c8-4016-866b-21c73206765a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288393 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-token\" (UniqueName: \"kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-token\") pod \"7a2a7348-14ea-449a-9988-99ff5f48506f\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288449 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-kube-api-access-kr7kj\" (UniqueName: \"kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-kube-api-access-kr7kj\") pod \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288562 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-certs" pod "44eda232-a5c8-4016-866b-21c73206765a" (UID: "44eda232-a5c8-4016-866b-21c73206765a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288641 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-certs" pod "5c37240b-0a43-4d38-a54e-47fdb0ff5c69" (UID: "5c37240b-0a43-4d38-a54e-47fdb0ff5c69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288674 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-credential-socket\") pod \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288710 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-certs\") pod \"7a2a7348-14ea-449a-9988-99ff5f48506f\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288740 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44eda232-a5c8-4016-866b-21c73206765a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eda232-a5c8-4016-866b-21c73206765a-istio-podinfo\") pod \"44eda232-a5c8-4016-866b-21c73206765a\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288785 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-credential-socket\") pod \"7a2a7348-14ea-449a-9988-99ff5f48506f\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288816 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-data\") pod \"7a2a7348-14ea-449a-9988-99ff5f48506f\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288855 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-token\" (UniqueName: \"kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-token\") pod \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\" (UID: \"5c37240b-0a43-4d38-a54e-47fdb0ff5c69\") " Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288933 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-data\") pod \"44eda232-a5c8-4016-866b-21c73206765a\" (UID: \"44eda232-a5c8-4016-866b-21c73206765a\") " Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288967 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-envoy\") pod \"7a2a7348-14ea-449a-9988-99ff5f48506f\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " Nov 03 19:14:22.289003 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.288996 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2a7348-14ea-449a-9988-99ff5f48506f-istiod-ca-cert" pod "7a2a7348-14ea-449a-9988-99ff5f48506f" (UID: "7a2a7348-14ea-449a-9988-99ff5f48506f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:22.289938 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.289085 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-kube-api-access-44pww\" (UniqueName: \"kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-kube-api-access-44pww\") pod \"7a2a7348-14ea-449a-9988-99ff5f48506f\" (UID: \"7a2a7348-14ea-449a-9988-99ff5f48506f\") " Nov 03 19:14:22.289938 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.289236 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-data" pod "5c37240b-0a43-4d38-a54e-47fdb0ff5c69" (UID: "5c37240b-0a43-4d38-a54e-47fdb0ff5c69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.290078 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.290050 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.290078 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.290077 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.290268 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.290093 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eda232-a5c8-4016-866b-21c73206765a-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.290268 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.290108 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.290268 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.290122 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2a7348-14ea-449a-9988-99ff5f48506f-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.290268 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.290136 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.290268 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.290150 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.290268 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.290165 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.290682 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.290369 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-socket" pod "5c37240b-0a43-4d38-a54e-47fdb0ff5c69" (UID: "5c37240b-0a43-4d38-a54e-47fdb0ff5c69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.290682 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.290602 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-credential-socket" pod "44eda232-a5c8-4016-866b-21c73206765a" (UID: "44eda232-a5c8-4016-866b-21c73206765a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.291705 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.291679 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-kube-api-access-z72xp" pod "44eda232-a5c8-4016-866b-21c73206765a" (UID: "44eda232-a5c8-4016-866b-21c73206765a"). InnerVolumeSpecName "kube-api-access-z72xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:22.291807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.291714 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-istio-token" pod "44eda232-a5c8-4016-866b-21c73206765a" (UID: "44eda232-a5c8-4016-866b-21c73206765a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:22.292026 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.292004 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-credential-socket" pod "7a2a7348-14ea-449a-9988-99ff5f48506f" (UID: "7a2a7348-14ea-449a-9988-99ff5f48506f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.292194 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.292176 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-data" pod "7a2a7348-14ea-449a-9988-99ff5f48506f" (UID: "7a2a7348-14ea-449a-9988-99ff5f48506f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.293012 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.292845 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-podinfo" pod "7a2a7348-14ea-449a-9988-99ff5f48506f" (UID: "7a2a7348-14ea-449a-9988-99ff5f48506f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:22.293113 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.293077 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-certs" pod "7a2a7348-14ea-449a-9988-99ff5f48506f" (UID: "7a2a7348-14ea-449a-9988-99ff5f48506f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.293113 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.293095 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-token" pod "5c37240b-0a43-4d38-a54e-47fdb0ff5c69" (UID: "5c37240b-0a43-4d38-a54e-47fdb0ff5c69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:22.293295 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.293273 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-data" pod "44eda232-a5c8-4016-866b-21c73206765a" (UID: "44eda232-a5c8-4016-866b-21c73206765a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.293396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.293307 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-credential-socket" pod "5c37240b-0a43-4d38-a54e-47fdb0ff5c69" (UID: "5c37240b-0a43-4d38-a54e-47fdb0ff5c69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.293651 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.293629 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-envoy" pod "44eda232-a5c8-4016-866b-21c73206765a" (UID: "44eda232-a5c8-4016-866b-21c73206765a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.293740 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.293686 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-kube-api-access-kr7kj" pod "5c37240b-0a43-4d38-a54e-47fdb0ff5c69" (UID: "5c37240b-0a43-4d38-a54e-47fdb0ff5c69"). InnerVolumeSpecName "kube-api-access-kr7kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:22.293971 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.293948 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-podinfo" pod "5c37240b-0a43-4d38-a54e-47fdb0ff5c69" (UID: "5c37240b-0a43-4d38-a54e-47fdb0ff5c69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:22.294284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.294249 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44eda232-a5c8-4016-866b-21c73206765a-istio-podinfo" pod "44eda232-a5c8-4016-866b-21c73206765a" (UID: "44eda232-a5c8-4016-866b-21c73206765a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:22.294420 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.294403 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-kube-api-access-44pww" pod "7a2a7348-14ea-449a-9988-99ff5f48506f" (UID: "7a2a7348-14ea-449a-9988-99ff5f48506f"). InnerVolumeSpecName "kube-api-access-44pww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:22.295110 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.295089 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-envoy" pod "5c37240b-0a43-4d38-a54e-47fdb0ff5c69" (UID: "5c37240b-0a43-4d38-a54e-47fdb0ff5c69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.295479 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.295456 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-envoy" pod "7a2a7348-14ea-449a-9988-99ff5f48506f" (UID: "7a2a7348-14ea-449a-9988-99ff5f48506f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:22.296113 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.296090 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-token" pod "7a2a7348-14ea-449a-9988-99ff5f48506f" (UID: "7a2a7348-14ea-449a-9988-99ff5f48506f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:22.308604 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.308575 5019 generic.go:356] "Generic (PLEG): container finished" podID="44eda232-a5c8-4016-866b-21c73206765a" containerID="87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998" exitCode=0 Nov 03 19:14:22.308730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.308616 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" Nov 03 19:14:22.308730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.308655 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" event={"ID":"44eda232-a5c8-4016-866b-21c73206765a","Type":"ContainerDied","Data":"87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998"} Nov 03 19:14:22.308730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.308717 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52" event={"ID":"44eda232-a5c8-4016-866b-21c73206765a","Type":"ContainerDied","Data":"180e8c7bf5b947aa13ed3dd74d9c7a65f2a68657a3281500bd1c87a1f3b6eca4"} Nov 03 19:14:22.308943 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.308748 5019 scope.go:122] "RemoveContainer" containerID="87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998" Nov 03 19:14:22.310446 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.310418 5019 generic.go:356] "Generic (PLEG): container finished" podID="7a2a7348-14ea-449a-9988-99ff5f48506f" containerID="57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9" exitCode=0 Nov 03 19:14:22.310530 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.310465 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" Nov 03 19:14:22.310620 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.310602 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" event={"ID":"7a2a7348-14ea-449a-9988-99ff5f48506f","Type":"ContainerDied","Data":"57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9"} Nov 03 19:14:22.310680 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.310644 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf" event={"ID":"7a2a7348-14ea-449a-9988-99ff5f48506f","Type":"ContainerDied","Data":"8b62c5d6ce914c7083ab1ac68fc63659e675f6fd815f3514b27dcadc71ddbe7f"} Nov 03 19:14:22.312022 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.312000 5019 generic.go:356] "Generic (PLEG): container finished" podID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" containerID="dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae" exitCode=0 Nov 03 19:14:22.312107 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.312081 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" event={"ID":"5c37240b-0a43-4d38-a54e-47fdb0ff5c69","Type":"ContainerDied","Data":"dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae"} Nov 03 19:14:22.312151 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.312115 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" event={"ID":"5c37240b-0a43-4d38-a54e-47fdb0ff5c69","Type":"ContainerDied","Data":"1e2f305007f955d6719ca1ed03bb52dbb0349dd685c4b46af7f11bfe539e129c"} Nov 03 19:14:22.312191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.312184 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp" Nov 03 19:14:22.343451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.337661 5019 scope.go:122] "RemoveContainer" containerID="87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998" Nov 03 19:14:22.343451 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:14:22.341059 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998\": container with ID starting with 87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998 not found: ID does not exist" containerID="87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998" Nov 03 19:14:22.343451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.341122 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998"} err="failed to get container status \"87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998\": rpc error: code = NotFound desc = could not find container \"87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998\": container with ID starting with 87ffe44f49c4792c4f062973925876c64ea5a79af912eb79ee6a70e3be15c998 not found: ID does not exist" Nov 03 19:14:22.343451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.341148 5019 scope.go:122] "RemoveContainer" containerID="57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9" Nov 03 19:14:22.354100 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.354074 5019 scope.go:122] "RemoveContainer" containerID="57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9" Nov 03 19:14:22.354540 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:14:22.354499 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9\": container with ID starting with 57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9 not found: ID does not exist" containerID="57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9" Nov 03 19:14:22.354657 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.354552 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9"} err="failed to get container status \"57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9\": rpc error: code = NotFound desc = could not find container \"57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9\": container with ID starting with 57756d8f7f6e502d6e7d6b006c38e92fa7139082f16cf1bac487f5f4ada061b9 not found: ID does not exist" Nov 03 19:14:22.354657 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.354580 5019 scope.go:122] "RemoveContainer" containerID="dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae" Nov 03 19:14:22.372744 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.372717 5019 scope.go:122] "RemoveContainer" containerID="dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae" Nov 03 19:14:22.373106 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:14:22.373075 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae\": container with ID starting with dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae not found: ID does not exist" containerID="dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae" Nov 03 19:14:22.373177 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.373108 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae"} err="failed to get container status \"dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae\": rpc error: code = NotFound desc = could not find container \"dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae\": container with ID starting with dca0113d14a97eeecf1b25b7594d9dadcbf4b59d6113a269d23a4b23684480ae not found: ID does not exist" Nov 03 19:14:22.392339 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392301 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392363 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z72xp\" (UniqueName: \"kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-kube-api-access-z72xp\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392379 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392388 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392398 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392406 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392415 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eda232-a5c8-4016-866b-21c73206765a-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392423 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392432 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr7kj\" (UniqueName: \"kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-kube-api-access-kr7kj\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392446 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392455 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392463 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eda232-a5c8-4016-866b-21c73206765a-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392473 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392481 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392489 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392497 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eda232-a5c8-4016-866b-21c73206765a-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392506 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2a7348-14ea-449a-9988-99ff5f48506f-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392521 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44pww\" (UniqueName: \"kubernetes.io/projected/7a2a7348-14ea-449a-9988-99ff5f48506f-kube-api-access-44pww\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.392678 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.392530 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c37240b-0a43-4d38-a54e-47fdb0ff5c69-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:22.718795 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.717604 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52"] Nov 03 19:14:22.725351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.723602 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-22g52"] Nov 03 19:14:22.820366 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.820295 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp"] Nov 03 19:14:22.824936 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.824910 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-bz6cp"] Nov 03 19:14:22.928223 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.928190 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf"] Nov 03 19:14:22.932531 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:22.932502 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-6dtgf"] Nov 03 19:14:24.204303 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:24.204269 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44eda232-a5c8-4016-866b-21c73206765a" path="/var/lib/kubelet/pods/44eda232-a5c8-4016-866b-21c73206765a/volumes" Nov 03 19:14:24.223279 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:24.221788 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" path="/var/lib/kubelet/pods/5c37240b-0a43-4d38-a54e-47fdb0ff5c69/volumes" Nov 03 19:14:24.223279 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:24.222605 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2a7348-14ea-449a-9988-99ff5f48506f" path="/var/lib/kubelet/pods/7a2a7348-14ea-449a-9988-99ff5f48506f/volumes" Nov 03 19:14:29.628692 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:29.628607 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:14:29.629161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:29.628880 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:14:35.468842 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:35.468810 5019 generic.go:356] "Generic (PLEG): container finished" podID="b8a00172-f381-4936-8800-18bdd9f428d6" containerID="6dbdeb05a98c74bcf6a1bf259446849ac996f006ca2d585294941a6e88a57709" exitCode=0 Nov 03 19:14:35.469406 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:35.468865 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" event={"ID":"b8a00172-f381-4936-8800-18bdd9f428d6","Type":"ContainerDied","Data":"6dbdeb05a98c74bcf6a1bf259446849ac996f006ca2d585294941a6e88a57709"} Nov 03 19:14:35.469488 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:35.469474 5019 scope.go:122] "RemoveContainer" containerID="6dbdeb05a98c74bcf6a1bf259446849ac996f006ca2d585294941a6e88a57709" Nov 03 19:14:35.471423 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:35.471390 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:14:35.471501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:35.471472 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:14:35.471540 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:35.471506 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:14:36.494412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.494368 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" event={"ID":"b8a00172-f381-4936-8800-18bdd9f428d6","Type":"ContainerStarted","Data":"c49a2befcdc89260f9c66d7f08768c6764fd11e29b05af0498e9940be276b775"} Nov 03 19:14:36.617046 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.613828 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:14:36.617046 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.614154 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:14:36.617046 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.614206 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 19:14:36.631494 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.630397 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c"] Nov 03 19:14:36.631703 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.631593 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" containerName="istio-proxy" Nov 03 19:14:36.631703 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.631613 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" containerName="istio-proxy" Nov 03 19:14:36.631703 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.631680 5019 cpu_manager.go:414] "containerMap: removing container" podUID="44eda232-a5c8-4016-866b-21c73206765a" containerName="istio-proxy" Nov 03 19:14:36.631703 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.631689 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44eda232-a5c8-4016-866b-21c73206765a" containerName="istio-proxy" Nov 03 19:14:36.631917 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.631710 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7a2a7348-14ea-449a-9988-99ff5f48506f" containerName="istio-proxy" Nov 03 19:14:36.631917 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.631718 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2a7348-14ea-449a-9988-99ff5f48506f" containerName="istio-proxy" Nov 03 19:14:36.631917 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.631887 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="44eda232-a5c8-4016-866b-21c73206765a" containerName="istio-proxy" Nov 03 19:14:36.631917 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.631902 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2a7348-14ea-449a-9988-99ff5f48506f" containerName="istio-proxy" Nov 03 19:14:36.632109 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.631922 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c37240b-0a43-4d38-a54e-47fdb0ff5c69" containerName="istio-proxy" Nov 03 19:14:36.636404 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.636385 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:36.687348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.686611 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-81a044-serving-cert\"" Nov 03 19:14:36.731758 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.731723 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c"] Nov 03 19:14:36.833711 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.827790 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:36.833711 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.827978 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:36.833711 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.828161 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b363559-7b87-4dde-a95b-8fe222208696-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:36.833711 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.828297 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qv44\" (UniqueName: \"kubernetes.io/projected/9b363559-7b87-4dde-a95b-8fe222208696-kube-api-access-2qv44\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:36.943695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.933356 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:36.943695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.933563 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b363559-7b87-4dde-a95b-8fe222208696-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:36.943695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.933698 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qv44\" (UniqueName: \"kubernetes.io/projected/9b363559-7b87-4dde-a95b-8fe222208696-kube-api-access-2qv44\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:36.943695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.933804 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:36.943695 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:14:36.934471 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-serving-cert: secret "maas-api-e2e-shared-b-w4-81a044-serving-cert" not found Nov 03 19:14:37.003380 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:36.957241 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b363559-7b87-4dde-a95b-8fe222208696-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:37.003380 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:14:36.984729 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-maas-api-tls podName:9b363559-7b87-4dde-a95b-8fe222208696 nodeName:}" failed. No retries permitted until 2026-11-03 19:14:37.484697113 +0000 UTC m=+1811.457258288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-maas-api-tls") pod "maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" (UID: "9b363559-7b87-4dde-a95b-8fe222208696") : secret "maas-api-e2e-shared-b-w4-81a044-serving-cert" not found Nov 03 19:14:37.094491 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:37.086969 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:37.094491 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:37.090083 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qv44\" (UniqueName: \"kubernetes.io/projected/9b363559-7b87-4dde-a95b-8fe222208696-kube-api-access-2qv44\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:37.526353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:37.509071 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:37.576350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:37.574428 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:37.654759 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:37.654712 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:14:37.655404 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:37.655369 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 19:14:37.852280 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:37.852201 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:38.348484 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:38.348450 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c"] Nov 03 19:14:38.349038 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:14:38.349013 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b363559_7b87_4dde_a95b_8fe222208696.slice/crio-7c4461a5ff378734e013ff40814056b3050247e6e392c3a40a8ab3258e9c46fd WatchSource:0}: Error finding container 7c4461a5ff378734e013ff40814056b3050247e6e392c3a40a8ab3258e9c46fd: Status 404 returned error can't find the container with id 7c4461a5ff378734e013ff40814056b3050247e6e392c3a40a8ab3258e9c46fd Nov 03 19:14:38.629437 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:38.627303 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:14:38.629437 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:38.627397 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 19:14:38.656633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:38.656594 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" event={"ID":"9b363559-7b87-4dde-a95b-8fe222208696","Type":"ContainerStarted","Data":"7c4461a5ff378734e013ff40814056b3050247e6e392c3a40a8ab3258e9c46fd"} Nov 03 19:14:39.615915 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:39.615744 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:14:39.615915 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:39.615802 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 19:14:39.712353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:39.708982 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" event={"ID":"9b363559-7b87-4dde-a95b-8fe222208696","Type":"ContainerStarted","Data":"c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d"} Nov 03 19:14:39.712353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:39.710286 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:39.841010 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:39.840947 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" podStartSLOduration=2.860078322 podStartE2EDuration="3.84092887s" totalImagesPullingTime="980.850548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:14:39.839252108 +0000 UTC m=+1813.811813308" watchObservedRunningTime="2026-11-03 19:14:39.84092887 +0000 UTC m=+1813.813490053" Nov 03 19:14:40.626580 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:40.625462 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:14:40.626580 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:40.625538 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 19:14:41.359208 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:14:41.359175 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:14:41.627409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:41.617425 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:14:41.627409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:41.617488 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 19:14:42.567689 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:42.567658 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:14:42.614558 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:42.614355 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:14:42.614864 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:42.614829 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 19:14:43.618196 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:43.618167 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:14:43.619342 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:43.619309 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:14:46.535436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.535402 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh"] Nov 03 19:14:46.535931 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.535662 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" containerID="cri-o://b38381705dfb80664400ebde74d7a72406aaa3fa89cb394395069f0bfd3829f3" gracePeriod=30 Nov 03 19:14:46.585820 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.585784 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb"] Nov 03 19:14:46.586159 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.586131 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" containerID="cri-o://ec81dcb3c4493590940f9f11fc39a2247ea25bc41cd46a8665cf2a4787344bea" gracePeriod=30 Nov 03 19:14:46.594220 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.594178 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8"] Nov 03 19:14:46.594464 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.594418 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" containerID="cri-o://d47b9bc25fc1102d91b3f4a49771bba0867b041c48db062fe8b2b48a641ba383" gracePeriod=30 Nov 03 19:14:46.650995 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.646374 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp"] Nov 03 19:14:46.650995 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.646718 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" containerID="cri-o://1545347abaded93fa19f298a96aff2c457b2c2eaaa7397b07f70d4abf58d296c" gracePeriod=30 Nov 03 19:14:46.659291 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.659260 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh"] Nov 03 19:14:46.659617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.659568 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" containerID="cri-o://3f6fc9666989d35f7ca27fa3d0f6ae15282ccbb3a2e2fb634693e20f774c07c0" gracePeriod=30 Nov 03 19:14:46.666738 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.666709 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p"] Nov 03 19:14:46.666943 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.666918 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" containerID="cri-o://e4f9609cdf4e853bcb33396638e197795c6c706d5fcfe83ba1fcfe708026a298" gracePeriod=30 Nov 03 19:14:46.683771 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.681005 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv"] Nov 03 19:14:46.683771 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.681247 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" containerID="cri-o://4e5858b0912ec3d7144bab6b112b6475f8f37dbe2415226757caf8f7974cedf3" gracePeriod=30 Nov 03 19:14:46.692723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.692678 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw"] Nov 03 19:14:46.692945 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.692915 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" containerID="cri-o://8f17092bd2660c8a2809dae05a62fcc51b760ee12368632d8b89d8670f562368" gracePeriod=30 Nov 03 19:14:46.843603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:46.843517 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:14:47.312032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:47.308688 5019 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": read tcp 10.132.0.2:53240->10.132.0.110:15021: read: connection reset by peer" start-of-body= Nov 03 19:14:47.312032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:47.308754 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": read tcp 10.132.0.2:53240->10.132.0.110:15021: read: connection reset by peer" Nov 03 19:14:47.929579 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:47.929540 5019 generic.go:356] "Generic (PLEG): container finished" podID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerID="3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94" exitCode=0 Nov 03 19:14:47.930016 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:47.929609 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" event={"ID":"39acdc92-a6a2-488c-b2c2-6a6133f0ce38","Type":"ContainerDied","Data":"3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94"} Nov 03 19:14:47.930117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:47.930102 5019 scope.go:122] "RemoveContainer" containerID="3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94" Nov 03 19:14:47.931981 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:47.931951 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:14:47.932066 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:47.932022 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:14:47.932066 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:47.932051 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:14:48.941855 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:48.941814 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" event={"ID":"39acdc92-a6a2-488c-b2c2-6a6133f0ce38","Type":"ContainerStarted","Data":"64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99"} Nov 03 19:14:49.691735 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:49.691696 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:14:49.691952 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:49.691820 5019 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 19:14:49.691952 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:49.691867 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 19:14:49.801871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:49.801823 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:14:49.802042 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:49.801905 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:14:50.503350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:50.503186 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:14:50.503350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:50.503269 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:14:50.691493 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:50.691456 5019 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 19:14:50.691785 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:50.691754 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 19:14:50.740692 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:50.740650 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:14:50.740896 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:50.740720 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:14:51.698427 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:51.698388 5019 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 19:14:51.699097 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:51.699064 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 19:14:52.432911 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.432872 5019 generic.go:356] "Generic (PLEG): container finished" podID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerID="ec81dcb3c4493590940f9f11fc39a2247ea25bc41cd46a8665cf2a4787344bea" exitCode=0 Nov 03 19:14:52.435424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.435312 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" event={"ID":"59721d1a-3083-4cd0-9f11-dd6c6c862e2c","Type":"ContainerDied","Data":"ec81dcb3c4493590940f9f11fc39a2247ea25bc41cd46a8665cf2a4787344bea"} Nov 03 19:14:52.440633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.440608 5019 generic.go:356] "Generic (PLEG): container finished" podID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerID="b38381705dfb80664400ebde74d7a72406aaa3fa89cb394395069f0bfd3829f3" exitCode=0 Nov 03 19:14:52.440899 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.440880 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" event={"ID":"5455b012-cb7c-4ec5-bdc4-076bb20f9eae","Type":"ContainerDied","Data":"b38381705dfb80664400ebde74d7a72406aaa3fa89cb394395069f0bfd3829f3"} Nov 03 19:14:52.441063 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.441049 5019 scope.go:122] "RemoveContainer" containerID="e24ec0ec767d393805c70f697a07daafbf651d3ceba903918f0549c90593b7b1" Nov 03 19:14:52.446680 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.446659 5019 generic.go:356] "Generic (PLEG): container finished" podID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerID="e4f9609cdf4e853bcb33396638e197795c6c706d5fcfe83ba1fcfe708026a298" exitCode=0 Nov 03 19:14:52.447558 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.447537 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" event={"ID":"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320","Type":"ContainerDied","Data":"e4f9609cdf4e853bcb33396638e197795c6c706d5fcfe83ba1fcfe708026a298"} Nov 03 19:14:52.450538 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.450518 5019 generic.go:356] "Generic (PLEG): container finished" podID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerID="4e5858b0912ec3d7144bab6b112b6475f8f37dbe2415226757caf8f7974cedf3" exitCode=0 Nov 03 19:14:52.450848 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.450828 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" event={"ID":"4c7e0ab3-5865-4566-a231-ad0b0c05810d","Type":"ContainerDied","Data":"4e5858b0912ec3d7144bab6b112b6475f8f37dbe2415226757caf8f7974cedf3"} Nov 03 19:14:52.493354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.485874 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:14:52.493354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.485957 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:14:52.561537 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.561491 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 19:14:52.561810 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.561765 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 19:14:52.655780 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.655746 5019 generic.go:356] "Generic (PLEG): container finished" podID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerID="d47b9bc25fc1102d91b3f4a49771bba0867b041c48db062fe8b2b48a641ba383" exitCode=0 Nov 03 19:14:52.656257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.656164 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" event={"ID":"fbb92e12-7aec-430a-9063-fbfe91123c44","Type":"ContainerDied","Data":"d47b9bc25fc1102d91b3f4a49771bba0867b041c48db062fe8b2b48a641ba383"} Nov 03 19:14:52.719863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.683641 5019 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 19:14:52.719863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.683708 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 19:14:52.726279 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.726058 5019 generic.go:356] "Generic (PLEG): container finished" podID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerID="3f6fc9666989d35f7ca27fa3d0f6ae15282ccbb3a2e2fb634693e20f774c07c0" exitCode=0 Nov 03 19:14:52.726279 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.726211 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" event={"ID":"887ebeb2-b7c1-45e4-98d2-a125ff72b729","Type":"ContainerDied","Data":"3f6fc9666989d35f7ca27fa3d0f6ae15282ccbb3a2e2fb634693e20f774c07c0"} Nov 03 19:14:52.729509 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.729320 5019 generic.go:356] "Generic (PLEG): container finished" podID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerID="8f17092bd2660c8a2809dae05a62fcc51b760ee12368632d8b89d8670f562368" exitCode=0 Nov 03 19:14:52.729509 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.729439 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" event={"ID":"dfc06a73-8b9e-48d9-b4b3-266d74b48f98","Type":"ContainerDied","Data":"8f17092bd2660c8a2809dae05a62fcc51b760ee12368632d8b89d8670f562368"} Nov 03 19:14:52.732611 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.732497 5019 generic.go:356] "Generic (PLEG): container finished" podID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerID="1545347abaded93fa19f298a96aff2c457b2c2eaaa7397b07f70d4abf58d296c" exitCode=0 Nov 03 19:14:52.732611 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.732537 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" event={"ID":"2389b6ad-dc9d-46e4-9380-3f5595fe53be","Type":"ContainerDied","Data":"1545347abaded93fa19f298a96aff2c457b2c2eaaa7397b07f70d4abf58d296c"} Nov 03 19:14:52.851200 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.851171 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 19:14:52.917614 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.917578 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:14:52.933560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.933526 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.970995 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-envoy\") pod \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971063 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-podinfo\") pod \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971129 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-credential-socket\") pod \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971171 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-certs\") pod \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971204 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-certs\") pod \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971230 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-data\") pod \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971289 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-credential-socket\") pod \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971318 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-token\" (UniqueName: \"kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-token\") pod \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971378 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-envoy\") pod \"fbb92e12-7aec-430a-9063-fbfe91123c44\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971433 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb92e12-7aec-430a-9063-fbfe91123c44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb92e12-7aec-430a-9063-fbfe91123c44-istiod-ca-cert\") pod \"fbb92e12-7aec-430a-9063-fbfe91123c44\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971575 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-data\") pod \"fbb92e12-7aec-430a-9063-fbfe91123c44\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971618 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-kube-api-access-2tprr\" (UniqueName: \"kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-kube-api-access-2tprr\") pod \"fbb92e12-7aec-430a-9063-fbfe91123c44\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971687 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-credential-socket\") pod \"fbb92e12-7aec-430a-9063-fbfe91123c44\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971773 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istiod-ca-cert\") pod \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971844 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-podinfo\") pod \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971871 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-token\" (UniqueName: \"kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-token\") pod \"fbb92e12-7aec-430a-9063-fbfe91123c44\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971906 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-kube-api-access-cls4g\" (UniqueName: \"kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-kube-api-access-cls4g\") pod \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971934 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-socket\") pod \"fbb92e12-7aec-430a-9063-fbfe91123c44\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.971969 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-podinfo\") pod \"fbb92e12-7aec-430a-9063-fbfe91123c44\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.972002 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-kube-api-access-qbxdm\" (UniqueName: \"kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-kube-api-access-qbxdm\") pod \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.972063 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-socket\") pod \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.972144 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istiod-ca-cert\") pod \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.972209 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-certs\") pod \"fbb92e12-7aec-430a-9063-fbfe91123c44\" (UID: \"fbb92e12-7aec-430a-9063-fbfe91123c44\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.972277 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-data\") pod \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.972349 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-token\" (UniqueName: \"kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-token\") pod \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.972384 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-envoy\") pod \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\" (UID: \"5455b012-cb7c-4ec5-bdc4-076bb20f9eae\") " Nov 03 19:14:52.975438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:52.972462 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-socket\") pod \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\" (UID: \"59721d1a-3083-4cd0-9f11-dd6c6c862e2c\") " Nov 03 19:14:53.044141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.015460 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-socket" pod "59721d1a-3083-4cd0-9f11-dd6c6c862e2c" (UID: "59721d1a-3083-4cd0-9f11-dd6c6c862e2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.044141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.020437 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-credential-socket" pod "fbb92e12-7aec-430a-9063-fbfe91123c44" (UID: "fbb92e12-7aec-430a-9063-fbfe91123c44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.044141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.027905 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-envoy" pod "59721d1a-3083-4cd0-9f11-dd6c6c862e2c" (UID: "59721d1a-3083-4cd0-9f11-dd6c6c862e2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.044141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.031292 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-credential-socket" pod "59721d1a-3083-4cd0-9f11-dd6c6c862e2c" (UID: "59721d1a-3083-4cd0-9f11-dd6c6c862e2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.044141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.031584 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-certs" pod "5455b012-cb7c-4ec5-bdc4-076bb20f9eae" (UID: "5455b012-cb7c-4ec5-bdc4-076bb20f9eae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.044141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.031795 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-certs" pod "59721d1a-3083-4cd0-9f11-dd6c6c862e2c" (UID: "59721d1a-3083-4cd0-9f11-dd6c6c862e2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.044141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.039311 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-credential-socket" pod "5455b012-cb7c-4ec5-bdc4-076bb20f9eae" (UID: "5455b012-cb7c-4ec5-bdc4-076bb20f9eae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.044141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.039793 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-certs" pod "fbb92e12-7aec-430a-9063-fbfe91123c44" (UID: "fbb92e12-7aec-430a-9063-fbfe91123c44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.044585 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.044394 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-socket" pod "5455b012-cb7c-4ec5-bdc4-076bb20f9eae" (UID: "5455b012-cb7c-4ec5-bdc4-076bb20f9eae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.045443 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.044870 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-kube-api-access-qbxdm" pod "59721d1a-3083-4cd0-9f11-dd6c6c862e2c" (UID: "59721d1a-3083-4cd0-9f11-dd6c6c862e2c"). InnerVolumeSpecName "kube-api-access-qbxdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.045443 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.045183 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istiod-ca-cert" pod "5455b012-cb7c-4ec5-bdc4-076bb20f9eae" (UID: "5455b012-cb7c-4ec5-bdc4-076bb20f9eae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:53.051126 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.047925 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istiod-ca-cert" pod "59721d1a-3083-4cd0-9f11-dd6c6c862e2c" (UID: "59721d1a-3083-4cd0-9f11-dd6c6c862e2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:53.051435 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.051403 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-data" pod "fbb92e12-7aec-430a-9063-fbfe91123c44" (UID: "fbb92e12-7aec-430a-9063-fbfe91123c44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.053429 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.051718 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-token" pod "5455b012-cb7c-4ec5-bdc4-076bb20f9eae" (UID: "5455b012-cb7c-4ec5-bdc4-076bb20f9eae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.053429 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.051782 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-envoy" pod "fbb92e12-7aec-430a-9063-fbfe91123c44" (UID: "fbb92e12-7aec-430a-9063-fbfe91123c44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.064349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.060831 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb92e12-7aec-430a-9063-fbfe91123c44-istiod-ca-cert" pod "fbb92e12-7aec-430a-9063-fbfe91123c44" (UID: "fbb92e12-7aec-430a-9063-fbfe91123c44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:53.064349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.061286 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-socket" pod "fbb92e12-7aec-430a-9063-fbfe91123c44" (UID: "fbb92e12-7aec-430a-9063-fbfe91123c44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.067119 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.066447 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-data" pod "59721d1a-3083-4cd0-9f11-dd6c6c862e2c" (UID: "59721d1a-3083-4cd0-9f11-dd6c6c862e2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.101349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.084456 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-podinfo" pod "5455b012-cb7c-4ec5-bdc4-076bb20f9eae" (UID: "5455b012-cb7c-4ec5-bdc4-076bb20f9eae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:53.112432 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.110294 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-data" pod "5455b012-cb7c-4ec5-bdc4-076bb20f9eae" (UID: "5455b012-cb7c-4ec5-bdc4-076bb20f9eae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115144 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115182 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115198 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115213 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115227 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115241 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115255 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb92e12-7aec-430a-9063-fbfe91123c44-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115268 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115281 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115293 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115306 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115319 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbxdm\" (UniqueName: \"kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-kube-api-access-qbxdm\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115351 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115365 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115377 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb92e12-7aec-430a-9063-fbfe91123c44-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115391 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115403 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115416 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115428 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.123353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.115441 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.244292 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.244205 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-podinfo" pod "fbb92e12-7aec-430a-9063-fbfe91123c44" (UID: "fbb92e12-7aec-430a-9063-fbfe91123c44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:53.244292 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.244234 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-kube-api-access-2tprr" pod "fbb92e12-7aec-430a-9063-fbfe91123c44" (UID: "fbb92e12-7aec-430a-9063-fbfe91123c44"). InnerVolumeSpecName "kube-api-access-2tprr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.246488 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.246462 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.246609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.246491 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tprr\" (UniqueName: \"kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-kube-api-access-2tprr\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.247644 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.247621 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-token" pod "59721d1a-3083-4cd0-9f11-dd6c6c862e2c" (UID: "59721d1a-3083-4cd0-9f11-dd6c6c862e2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.247763 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.247746 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-podinfo" pod "59721d1a-3083-4cd0-9f11-dd6c6c862e2c" (UID: "59721d1a-3083-4cd0-9f11-dd6c6c862e2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:53.250958 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.250635 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-kube-api-access-cls4g" pod "5455b012-cb7c-4ec5-bdc4-076bb20f9eae" (UID: "5455b012-cb7c-4ec5-bdc4-076bb20f9eae"). InnerVolumeSpecName "kube-api-access-cls4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.256685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.256651 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-token" pod "fbb92e12-7aec-430a-9063-fbfe91123c44" (UID: "fbb92e12-7aec-430a-9063-fbfe91123c44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.257060 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.256857 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-envoy" pod "5455b012-cb7c-4ec5-bdc4-076bb20f9eae" (UID: "5455b012-cb7c-4ec5-bdc4-076bb20f9eae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.348784 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.347617 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:14:53.348784 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.348219 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.348784 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.348246 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb92e12-7aec-430a-9063-fbfe91123c44-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.348784 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.348260 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cls4g\" (UniqueName: \"kubernetes.io/projected/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-kube-api-access-cls4g\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.348784 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.348275 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59721d1a-3083-4cd0-9f11-dd6c6c862e2c-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.348784 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.348290 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5455b012-cb7c-4ec5-bdc4-076bb20f9eae-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.350919 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.350896 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:14:53.353559 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.353535 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:14:53.356929 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.356906 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:14:53.366529 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.366508 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:14:53.449481 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449443 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-kube-api-access-h49zf\" (UniqueName: \"kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-kube-api-access-h49zf\") pod \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " Nov 03 19:14:53.449668 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449497 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-certs\") pod \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " Nov 03 19:14:53.449668 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449527 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-data\") pod \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " Nov 03 19:14:53.449668 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449565 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-podinfo\") pod \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " Nov 03 19:14:53.449668 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449594 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-envoy\") pod \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " Nov 03 19:14:53.449668 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449626 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-certs\") pod \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " Nov 03 19:14:53.449668 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449649 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-token\" (UniqueName: \"kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-token\") pod \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " Nov 03 19:14:53.449983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449688 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istiod-ca-cert\") pod \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " Nov 03 19:14:53.449983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449748 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-podinfo\") pod \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " Nov 03 19:14:53.449983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449787 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istiod-ca-cert\") pod \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " Nov 03 19:14:53.449983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449819 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istiod-ca-cert\") pod \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " Nov 03 19:14:53.449983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449855 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-kube-api-access-jzfbx\" (UniqueName: \"kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-kube-api-access-jzfbx\") pod \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " Nov 03 19:14:53.449983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449900 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-certs\") pod \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " Nov 03 19:14:53.449983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449936 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-credential-socket\") pod \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " Nov 03 19:14:53.449983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.449968 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istiod-ca-cert\") pod \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " Nov 03 19:14:53.450403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450003 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-envoy\") pod \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " Nov 03 19:14:53.450403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450051 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-envoy\") pod \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " Nov 03 19:14:53.450403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450108 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-podinfo\") pod \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " Nov 03 19:14:53.450403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450225 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-data\") pod \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " Nov 03 19:14:53.450403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450253 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-socket\") pod \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " Nov 03 19:14:53.450403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450285 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-token\" (UniqueName: \"kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-token\") pod \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " Nov 03 19:14:53.450708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450415 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-credential-socket\") pod \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " Nov 03 19:14:53.450708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450500 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-envoy\") pod \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " Nov 03 19:14:53.450708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450566 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-data\") pod \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " Nov 03 19:14:53.450708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450598 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-kube-api-access-hm6kb\" (UniqueName: \"kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-kube-api-access-hm6kb\") pod \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " Nov 03 19:14:53.450708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450653 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-credential-socket\") pod \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " Nov 03 19:14:53.450972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450707 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-token\" (UniqueName: \"kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-token\") pod \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " Nov 03 19:14:53.450972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450747 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-certs\") pod \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " Nov 03 19:14:53.450972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450787 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-certs\") pod \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " Nov 03 19:14:53.450972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450842 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-data\") pod \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " Nov 03 19:14:53.450972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450882 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-socket\") pod \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " Nov 03 19:14:53.450972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450924 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-token\" (UniqueName: \"kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-token\") pod \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " Nov 03 19:14:53.450972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.450961 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-credential-socket\") pod \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " Nov 03 19:14:53.451353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.451005 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istiod-ca-cert\") pod \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " Nov 03 19:14:53.451353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.451045 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-envoy\") pod \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " Nov 03 19:14:53.451353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.451094 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-socket\") pod \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " Nov 03 19:14:53.451353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.451196 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-credential-socket\") pod \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\" (UID: \"dfc06a73-8b9e-48d9-b4b3-266d74b48f98\") " Nov 03 19:14:53.451353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.451281 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-kube-api-access-6m6pj\" (UniqueName: \"kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-kube-api-access-6m6pj\") pod \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\" (UID: \"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320\") " Nov 03 19:14:53.451621 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.451366 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-podinfo\") pod \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " Nov 03 19:14:53.451621 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.451411 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-socket\") pod \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " Nov 03 19:14:53.451621 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.451446 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-data\") pod \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " Nov 03 19:14:53.451621 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.451536 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-socket\") pod \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\" (UID: \"2389b6ad-dc9d-46e4-9380-3f5595fe53be\") " Nov 03 19:14:53.451621 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.451613 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-kube-api-access-lknf7\" (UniqueName: \"kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-kube-api-access-lknf7\") pod \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " Nov 03 19:14:53.451859 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.451644 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-podinfo\") pod \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\" (UID: \"887ebeb2-b7c1-45e4-98d2-a125ff72b729\") " Nov 03 19:14:53.451859 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.451734 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-token\" (UniqueName: \"kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-token\") pod \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\" (UID: \"4c7e0ab3-5865-4566-a231-ad0b0c05810d\") " Nov 03 19:14:53.452683 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.452653 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-credential-socket" pod "2389b6ad-dc9d-46e4-9380-3f5595fe53be" (UID: "2389b6ad-dc9d-46e4-9380-3f5595fe53be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.453143 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.453114 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istiod-ca-cert" pod "2389b6ad-dc9d-46e4-9380-3f5595fe53be" (UID: "2389b6ad-dc9d-46e4-9380-3f5595fe53be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:53.453398 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.453372 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-credential-socket" pod "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" (UID: "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.453502 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.453456 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-certs" pod "2389b6ad-dc9d-46e4-9380-3f5595fe53be" (UID: "2389b6ad-dc9d-46e4-9380-3f5595fe53be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.454210 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.453623 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-data" pod "4c7e0ab3-5865-4566-a231-ad0b0c05810d" (UID: "4c7e0ab3-5865-4566-a231-ad0b0c05810d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.454210 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.453697 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-data" pod "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" (UID: "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.454210 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.453707 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istiod-ca-cert" pod "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" (UID: "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:53.456071 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.454708 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-credential-socket" pod "887ebeb2-b7c1-45e4-98d2-a125ff72b729" (UID: "887ebeb2-b7c1-45e4-98d2-a125ff72b729"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.456071 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.454947 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-certs" pod "887ebeb2-b7c1-45e4-98d2-a125ff72b729" (UID: "887ebeb2-b7c1-45e4-98d2-a125ff72b729"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.456071 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.455467 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-data" pod "dfc06a73-8b9e-48d9-b4b3-266d74b48f98" (UID: "dfc06a73-8b9e-48d9-b4b3-266d74b48f98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.456868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.456569 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-certs" pod "dfc06a73-8b9e-48d9-b4b3-266d74b48f98" (UID: "dfc06a73-8b9e-48d9-b4b3-266d74b48f98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.456868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.456596 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-certs" pod "4c7e0ab3-5865-4566-a231-ad0b0c05810d" (UID: "4c7e0ab3-5865-4566-a231-ad0b0c05810d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.456868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.456809 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-data" pod "2389b6ad-dc9d-46e4-9380-3f5595fe53be" (UID: "2389b6ad-dc9d-46e4-9380-3f5595fe53be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.457083 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.457005 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-socket" pod "dfc06a73-8b9e-48d9-b4b3-266d74b48f98" (UID: "dfc06a73-8b9e-48d9-b4b3-266d74b48f98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.457140 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.457086 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-credential-socket" pod "dfc06a73-8b9e-48d9-b4b3-266d74b48f98" (UID: "dfc06a73-8b9e-48d9-b4b3-266d74b48f98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.458673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.457469 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-socket" pod "4c7e0ab3-5865-4566-a231-ad0b0c05810d" (UID: "4c7e0ab3-5865-4566-a231-ad0b0c05810d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.458673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.457723 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-token" pod "4c7e0ab3-5865-4566-a231-ad0b0c05810d" (UID: "4c7e0ab3-5865-4566-a231-ad0b0c05810d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.458673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.457931 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-data" pod "887ebeb2-b7c1-45e4-98d2-a125ff72b729" (UID: "887ebeb2-b7c1-45e4-98d2-a125ff72b729"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.458673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.458166 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istiod-ca-cert" pod "dfc06a73-8b9e-48d9-b4b3-266d74b48f98" (UID: "dfc06a73-8b9e-48d9-b4b3-266d74b48f98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:53.458673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.458222 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istiod-ca-cert" pod "887ebeb2-b7c1-45e4-98d2-a125ff72b729" (UID: "887ebeb2-b7c1-45e4-98d2-a125ff72b729"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:53.458673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.458368 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-socket" pod "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" (UID: "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.460125 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.458766 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istiod-ca-cert" pod "4c7e0ab3-5865-4566-a231-ad0b0c05810d" (UID: "4c7e0ab3-5865-4566-a231-ad0b0c05810d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:53.460125 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.458823 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-credential-socket" pod "4c7e0ab3-5865-4566-a231-ad0b0c05810d" (UID: "4c7e0ab3-5865-4566-a231-ad0b0c05810d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.475238 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.474984 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-socket" pod "887ebeb2-b7c1-45e4-98d2-a125ff72b729" (UID: "887ebeb2-b7c1-45e4-98d2-a125ff72b729"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.475238 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.475153 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-token" pod "2389b6ad-dc9d-46e4-9380-3f5595fe53be" (UID: "2389b6ad-dc9d-46e4-9380-3f5595fe53be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.475238 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.475209 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-envoy" pod "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" (UID: "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.476056 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.475790 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-podinfo" pod "887ebeb2-b7c1-45e4-98d2-a125ff72b729" (UID: "887ebeb2-b7c1-45e4-98d2-a125ff72b729"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:53.476056 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.475859 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-envoy" pod "4c7e0ab3-5865-4566-a231-ad0b0c05810d" (UID: "4c7e0ab3-5865-4566-a231-ad0b0c05810d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.500420 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.477539 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-socket" pod "2389b6ad-dc9d-46e4-9380-3f5595fe53be" (UID: "2389b6ad-dc9d-46e4-9380-3f5595fe53be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.500420 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.478782 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-podinfo" pod "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" (UID: "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:53.500420 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.478838 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-envoy" pod "2389b6ad-dc9d-46e4-9380-3f5595fe53be" (UID: "2389b6ad-dc9d-46e4-9380-3f5595fe53be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.500420 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.489019 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-certs" pod "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" (UID: "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.513354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.505514 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-kube-api-access-lknf7" pod "887ebeb2-b7c1-45e4-98d2-a125ff72b729" (UID: "887ebeb2-b7c1-45e4-98d2-a125ff72b729"). InnerVolumeSpecName "kube-api-access-lknf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.533474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.522559 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-kube-api-access-6m6pj" pod "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" (UID: "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320"). InnerVolumeSpecName "kube-api-access-6m6pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.533474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.526533 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-token" pod "887ebeb2-b7c1-45e4-98d2-a125ff72b729" (UID: "887ebeb2-b7c1-45e4-98d2-a125ff72b729"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.533474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.530647 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-kube-api-access-jzfbx" pod "4c7e0ab3-5865-4566-a231-ad0b0c05810d" (UID: "4c7e0ab3-5865-4566-a231-ad0b0c05810d"). InnerVolumeSpecName "kube-api-access-jzfbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.533474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.531452 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-envoy" pod "dfc06a73-8b9e-48d9-b4b3-266d74b48f98" (UID: "dfc06a73-8b9e-48d9-b4b3-266d74b48f98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.533474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.531524 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-envoy" pod "887ebeb2-b7c1-45e4-98d2-a125ff72b729" (UID: "887ebeb2-b7c1-45e4-98d2-a125ff72b729"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:53.533474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.531580 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-podinfo" pod "2389b6ad-dc9d-46e4-9380-3f5595fe53be" (UID: "2389b6ad-dc9d-46e4-9380-3f5595fe53be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:53.533474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.531632 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-podinfo" pod "dfc06a73-8b9e-48d9-b4b3-266d74b48f98" (UID: "dfc06a73-8b9e-48d9-b4b3-266d74b48f98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:53.533474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.531708 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-token" pod "dfc06a73-8b9e-48d9-b4b3-266d74b48f98" (UID: "dfc06a73-8b9e-48d9-b4b3-266d74b48f98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.534035 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.533522 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-kube-api-access-h49zf" pod "2389b6ad-dc9d-46e4-9380-3f5595fe53be" (UID: "2389b6ad-dc9d-46e4-9380-3f5595fe53be"). InnerVolumeSpecName "kube-api-access-h49zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.539448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.535656 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-podinfo" pod "4c7e0ab3-5865-4566-a231-ad0b0c05810d" (UID: "4c7e0ab3-5865-4566-a231-ad0b0c05810d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:53.539448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.537493 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-kube-api-access-hm6kb" pod "dfc06a73-8b9e-48d9-b4b3-266d74b48f98" (UID: "dfc06a73-8b9e-48d9-b4b3-266d74b48f98"). InnerVolumeSpecName "kube-api-access-hm6kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.540882 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.540814 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-token" pod "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" (UID: "7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.553889 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.553914 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h49zf\" (UniqueName: \"kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-kube-api-access-h49zf\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.553925 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.553934 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.553943 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.553951 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.553962 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.553975 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.553988 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554000 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554012 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554025 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554039 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzfbx\" (UniqueName: \"kubernetes.io/projected/4c7e0ab3-5865-4566-a231-ad0b0c05810d-kube-api-access-jzfbx\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554053 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554069 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554082 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554096 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554110 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554124 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554139 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554153 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554167 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554181 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554195 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554210 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554224 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm6kb\" (UniqueName: \"kubernetes.io/projected/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-kube-api-access-hm6kb\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554239 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554255 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554268 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554283 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554297 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554310 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554338 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554354 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554367 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554381 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554394 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554408 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc06a73-8b9e-48d9-b4b3-266d74b48f98-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554424 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m6pj\" (UniqueName: \"kubernetes.io/projected/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320-kube-api-access-6m6pj\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554439 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7e0ab3-5865-4566-a231-ad0b0c05810d-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554453 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554467 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554481 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2389b6ad-dc9d-46e4-9380-3f5595fe53be-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554494 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lknf7\" (UniqueName: \"kubernetes.io/projected/887ebeb2-b7c1-45e4-98d2-a125ff72b729-kube-api-access-lknf7\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.555176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.554508 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887ebeb2-b7c1-45e4-98d2-a125ff72b729-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:53.681882 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.681441 5019 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 19:14:53.681882 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.681503 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 19:14:53.825213 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.824559 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" event={"ID":"7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320","Type":"ContainerDied","Data":"5a6068ba105e7a5b4a81e8365494771a7da813aee9dedda2b1e42528b2f957ca"} Nov 03 19:14:53.825213 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.824670 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p" Nov 03 19:14:53.825213 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.824709 5019 scope.go:122] "RemoveContainer" containerID="e4f9609cdf4e853bcb33396638e197795c6c706d5fcfe83ba1fcfe708026a298" Nov 03 19:14:53.827841 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.827824 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" Nov 03 19:14:53.828090 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.827860 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv" event={"ID":"4c7e0ab3-5865-4566-a231-ad0b0c05810d","Type":"ContainerDied","Data":"0a21e82e84962b40c776f6e7df80bb22be3f92bd37baeadff22d09015d18b27b"} Nov 03 19:14:53.830562 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.830544 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" Nov 03 19:14:53.830707 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.830582 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8" event={"ID":"fbb92e12-7aec-430a-9063-fbfe91123c44","Type":"ContainerDied","Data":"b024c4c1242a0acd590a84d8c1d05d3522cec85fd363e62e5de626d464bef92c"} Nov 03 19:14:53.833014 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.832981 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" event={"ID":"887ebeb2-b7c1-45e4-98d2-a125ff72b729","Type":"ContainerDied","Data":"db2a648d9e8b112cc141044585b56b17dad0309e33abbce0fe03d8ba5af40bea"} Nov 03 19:14:53.833113 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.833071 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh" Nov 03 19:14:53.850800 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.849821 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" event={"ID":"dfc06a73-8b9e-48d9-b4b3-266d74b48f98","Type":"ContainerDied","Data":"4d00f014f1b7830188201432b9d747a01184fc90ed6c4996022abee612ebf9f0"} Nov 03 19:14:53.850800 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.849950 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw" Nov 03 19:14:53.859072 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.856897 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:14:53.859072 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.856952 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:14:53.924822 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.923530 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" Nov 03 19:14:53.924822 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.923609 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp" event={"ID":"2389b6ad-dc9d-46e4-9380-3f5595fe53be","Type":"ContainerDied","Data":"fe3840178bc6e01bf774b768ba9bef30ab69285ef487fc8b7d1424b71fd2d714"} Nov 03 19:14:53.926938 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.926921 5019 scope.go:122] "RemoveContainer" containerID="4e5858b0912ec3d7144bab6b112b6475f8f37dbe2415226757caf8f7974cedf3" Nov 03 19:14:53.932001 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.931622 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" Nov 03 19:14:53.932001 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.931625 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb" event={"ID":"59721d1a-3083-4cd0-9f11-dd6c6c862e2c","Type":"ContainerDied","Data":"cb9e0a12cc81b1591d4ec4c062ced8a4ec268b98f16e53f84cc0061c3a295537"} Nov 03 19:14:53.939271 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.933453 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" event={"ID":"5455b012-cb7c-4ec5-bdc4-076bb20f9eae","Type":"ContainerDied","Data":"75e48f5965ff8e14f317ed81ee898f89b13acc25c7d43161407ce8de77955113"} Nov 03 19:14:53.939271 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.933539 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh" Nov 03 19:14:53.948304 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.948285 5019 scope.go:122] "RemoveContainer" containerID="d47b9bc25fc1102d91b3f4a49771bba0867b041c48db062fe8b2b48a641ba383" Nov 03 19:14:53.995819 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:53.965118 5019 scope.go:122] "RemoveContainer" containerID="3f6fc9666989d35f7ca27fa3d0f6ae15282ccbb3a2e2fb634693e20f774c07c0" Nov 03 19:14:54.043267 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.043236 5019 scope.go:122] "RemoveContainer" containerID="8f17092bd2660c8a2809dae05a62fcc51b760ee12368632d8b89d8670f562368" Nov 03 19:14:54.055752 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.055729 5019 scope.go:122] "RemoveContainer" containerID="1545347abaded93fa19f298a96aff2c457b2c2eaaa7397b07f70d4abf58d296c" Nov 03 19:14:54.119027 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.119000 5019 scope.go:122] "RemoveContainer" containerID="ec81dcb3c4493590940f9f11fc39a2247ea25bc41cd46a8665cf2a4787344bea" Nov 03 19:14:54.131443 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.131394 5019 scope.go:122] "RemoveContainer" containerID="b38381705dfb80664400ebde74d7a72406aaa3fa89cb394395069f0bfd3829f3" Nov 03 19:14:54.231631 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.231592 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh"] Nov 03 19:14:54.235675 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.235645 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-m2jvh"] Nov 03 19:14:54.318556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.318517 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb"] Nov 03 19:14:54.322278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.322254 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-d26kb"] Nov 03 19:14:54.446428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.446394 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw"] Nov 03 19:14:54.450582 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.450550 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-5r4xw"] Nov 03 19:14:54.552220 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.552183 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv"] Nov 03 19:14:54.556356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.556312 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6xbvv"] Nov 03 19:14:54.658718 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.658685 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8"] Nov 03 19:14:54.687781 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.687741 5019 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 19:14:54.688049 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.688014 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 19:14:54.693905 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.693876 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-ljfs8"] Nov 03 19:14:54.794660 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.794569 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp"] Nov 03 19:14:54.801320 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.801290 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-h6kvp"] Nov 03 19:14:54.907308 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.907268 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh"] Nov 03 19:14:54.907308 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.907345 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-jk9rh"] Nov 03 19:14:54.997188 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:54.997153 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p"] Nov 03 19:14:55.000965 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:55.000937 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-6vs8p"] Nov 03 19:14:55.690913 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:55.690878 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:14:55.691268 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:55.691248 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:14:55.692038 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:55.692018 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:14:56.153666 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:56.153580 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" path="/var/lib/kubelet/pods/2389b6ad-dc9d-46e4-9380-3f5595fe53be/volumes" Nov 03 19:14:56.154298 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:56.154278 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" path="/var/lib/kubelet/pods/4c7e0ab3-5865-4566-a231-ad0b0c05810d/volumes" Nov 03 19:14:56.154842 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:56.154821 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" path="/var/lib/kubelet/pods/5455b012-cb7c-4ec5-bdc4-076bb20f9eae/volumes" Nov 03 19:14:56.155708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:56.155685 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" path="/var/lib/kubelet/pods/59721d1a-3083-4cd0-9f11-dd6c6c862e2c/volumes" Nov 03 19:14:56.156198 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:56.156178 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" path="/var/lib/kubelet/pods/7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320/volumes" Nov 03 19:14:56.156728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:56.156708 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" path="/var/lib/kubelet/pods/887ebeb2-b7c1-45e4-98d2-a125ff72b729/volumes" Nov 03 19:14:56.157520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:56.157499 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" path="/var/lib/kubelet/pods/dfc06a73-8b9e-48d9-b4b3-266d74b48f98/volumes" Nov 03 19:14:56.157997 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:14:56.157978 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" path="/var/lib/kubelet/pods/fbb92e12-7aec-430a-9063-fbfe91123c44/volumes" Nov 03 19:15:03.052826 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:03.052795 5019 generic.go:356] "Generic (PLEG): container finished" podID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerID="b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1" exitCode=0 Nov 03 19:15:03.053413 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:03.052872 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" event={"ID":"c0cdf3bc-673f-4fd1-bb49-92b3e4994229","Type":"ContainerDied","Data":"b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1"} Nov 03 19:15:03.053519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:03.053499 5019 scope.go:122] "RemoveContainer" containerID="b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1" Nov 03 19:15:03.055148 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:03.055118 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:03.055237 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:03.055188 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:03.055237 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:03.055222 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:04.088957 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:04.088918 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" event={"ID":"c0cdf3bc-673f-4fd1-bb49-92b3e4994229","Type":"ContainerStarted","Data":"b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb"} Nov 03 19:15:04.221849 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:04.221818 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:15:04.223303 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:04.223275 5019 patch_prober.go:29] interesting pod/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 19:15:04.223455 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:04.223351 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 19:15:05.150222 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:05.150192 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:15:05.221995 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:05.221961 5019 patch_prober.go:29] interesting pod/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 19:15:05.222159 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:05.222013 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 19:15:06.221713 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:06.221629 5019 patch_prober.go:29] interesting pod/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 19:15:06.221713 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:06.221693 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 19:15:07.221393 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:07.221362 5019 patch_prober.go:29] interesting pod/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 19:15:07.221576 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:07.221414 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 19:15:08.221962 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:08.221931 5019 patch_prober.go:29] interesting pod/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 19:15:08.222383 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:08.221983 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 19:15:09.291241 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:09.291205 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:15:09.292462 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:09.292435 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:15:10.655362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:10.655007 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz"] Nov 03 19:15:10.655362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:10.655260 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" podUID="65005b3c-db12-4ddc-ab6e-2a4261ee6b69" containerName="maas-api" containerID="cri-o://047e009872e0ada895ce266833d3524ebc5399a914ff3571d7701d915d15a2f0" gracePeriod=30 Nov 03 19:15:11.302408 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.300652 5019 generic.go:356] "Generic (PLEG): container finished" podID="65005b3c-db12-4ddc-ab6e-2a4261ee6b69" containerID="047e009872e0ada895ce266833d3524ebc5399a914ff3571d7701d915d15a2f0" exitCode=0 Nov 03 19:15:11.302408 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.300757 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" event={"ID":"65005b3c-db12-4ddc-ab6e-2a4261ee6b69","Type":"ContainerDied","Data":"047e009872e0ada895ce266833d3524ebc5399a914ff3571d7701d915d15a2f0"} Nov 03 19:15:11.434541 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:15:11.434502 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:15:11.574377 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.574291 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:15:11.699349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.696930 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-metrics-certs\") pod \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " Nov 03 19:15:11.699349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.697032 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-kube-api-access-f79pg\" (UniqueName: \"kubernetes.io/projected/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-kube-api-access-f79pg\") pod \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " Nov 03 19:15:11.699349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.697116 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-maas-api-tls\") pod \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " Nov 03 19:15:11.699349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.697174 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-openshift-service-ca\") pod \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\" (UID: \"65005b3c-db12-4ddc-ab6e-2a4261ee6b69\") " Nov 03 19:15:11.726349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.724554 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-openshift-service-ca" pod "65005b3c-db12-4ddc-ab6e-2a4261ee6b69" (UID: "65005b3c-db12-4ddc-ab6e-2a4261ee6b69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:11.737629 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.737532 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-maas-api-tls" pod "65005b3c-db12-4ddc-ab6e-2a4261ee6b69" (UID: "65005b3c-db12-4ddc-ab6e-2a4261ee6b69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:11.755467 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.751981 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-metrics-certs" pod "65005b3c-db12-4ddc-ab6e-2a4261ee6b69" (UID: "65005b3c-db12-4ddc-ab6e-2a4261ee6b69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:11.757226 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.757196 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-kube-api-access-f79pg" pod "65005b3c-db12-4ddc-ab6e-2a4261ee6b69" (UID: "65005b3c-db12-4ddc-ab6e-2a4261ee6b69"). InnerVolumeSpecName "kube-api-access-f79pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:11.799676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.799579 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:11.799676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.799617 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f79pg\" (UniqueName: \"kubernetes.io/projected/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-kube-api-access-f79pg\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:11.799676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.799632 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:11.799676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:11.799646 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65005b3c-db12-4ddc-ab6e-2a4261ee6b69-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:12.310225 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.310192 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" Nov 03 19:15:12.310421 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.310229 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz" event={"ID":"65005b3c-db12-4ddc-ab6e-2a4261ee6b69","Type":"ContainerDied","Data":"08b9fcd5a21be03ea4a9790aee17d96c2cbfcd0bf545354e2d420be91af7b335"} Nov 03 19:15:12.310421 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.310287 5019 scope.go:122] "RemoveContainer" containerID="047e009872e0ada895ce266833d3524ebc5399a914ff3571d7701d915d15a2f0" Nov 03 19:15:12.467607 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.467572 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz"] Nov 03 19:15:12.514373 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.514306 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5b012841-7c4f66646-vbcnz"] Nov 03 19:15:12.517943 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.517917 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5"] Nov 03 19:15:12.518754 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518727 5019 cpu_manager.go:414] "containerMap: removing container" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" Nov 03 19:15:12.518754 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518754 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518774 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518783 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518810 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518817 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518837 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518843 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518856 5019 cpu_manager.go:414] "containerMap: removing container" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518862 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518889 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518895 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518906 5019 cpu_manager.go:414] "containerMap: removing container" podUID="65005b3c-db12-4ddc-ab6e-2a4261ee6b69" containerName="maas-api" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518912 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65005b3c-db12-4ddc-ab6e-2a4261ee6b69" containerName="maas-api" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518933 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" Nov 03 19:15:12.518940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518939 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" Nov 03 19:15:12.519556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518958 5019 cpu_manager.go:414] "containerMap: removing container" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" Nov 03 19:15:12.519556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.518964 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" Nov 03 19:15:12.519556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.519107 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7e0ab3-5865-4566-a231-ad0b0c05810d" containerName="istio-proxy" Nov 03 19:15:12.519556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.519118 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb92e12-7aec-430a-9063-fbfe91123c44" containerName="istio-proxy" Nov 03 19:15:12.519556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.519132 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="59721d1a-3083-4cd0-9f11-dd6c6c862e2c" containerName="istio-proxy" Nov 03 19:15:12.519556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.519141 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" Nov 03 19:15:12.519556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.519152 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2389b6ad-dc9d-46e4-9380-3f5595fe53be" containerName="istio-proxy" Nov 03 19:15:12.519556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.519163 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" Nov 03 19:15:12.519556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.519172 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc06a73-8b9e-48d9-b4b3-266d74b48f98" containerName="istio-proxy" Nov 03 19:15:12.519556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.519182 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="887ebeb2-b7c1-45e4-98d2-a125ff72b729" containerName="istio-proxy" Nov 03 19:15:12.519556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.519193 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d10eecf-0ad3-4fc5-a2d3-eeeebe6e4320" containerName="istio-proxy" Nov 03 19:15:12.519556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.519200 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="65005b3c-db12-4ddc-ab6e-2a4261ee6b69" containerName="maas-api" Nov 03 19:15:12.524499 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.524477 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:12.574146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.574118 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bbccf3-serving-cert\"" Nov 03 19:15:12.589196 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.589165 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5"] Nov 03 19:15:12.647607 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.647568 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d508cbeb-a7cf-464f-88d4-b3374365e690-openshift-service-ca\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:12.647756 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.647726 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-maas-api-tls\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:12.647893 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.647876 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n82rn\" (UniqueName: \"kubernetes.io/projected/d508cbeb-a7cf-464f-88d4-b3374365e690-kube-api-access-n82rn\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:12.648055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.648037 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-metrics-certs\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:12.749357 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.749255 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-metrics-certs\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:12.749837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.749385 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d508cbeb-a7cf-464f-88d4-b3374365e690-openshift-service-ca\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:12.749837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.749456 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-maas-api-tls\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:12.749837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.749553 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n82rn\" (UniqueName: \"kubernetes.io/projected/d508cbeb-a7cf-464f-88d4-b3374365e690-kube-api-access-n82rn\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:12.749837 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:15:12.749608 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-serving-cert: secret "maas-api-e2e-api-a-bbccf3-serving-cert" not found Nov 03 19:15:12.749837 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:15:12.749684 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-maas-api-tls podName:d508cbeb-a7cf-464f-88d4-b3374365e690 nodeName:}" failed. No retries permitted until 2026-11-03 19:15:13.249662355 +0000 UTC m=+1847.222223529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-maas-api-tls") pod "maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" (UID: "d508cbeb-a7cf-464f-88d4-b3374365e690") : secret "maas-api-e2e-api-a-bbccf3-serving-cert" not found Nov 03 19:15:12.750216 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.750188 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d508cbeb-a7cf-464f-88d4-b3374365e690-openshift-service-ca\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:12.752087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.752068 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-metrics-certs\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:12.834360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:12.831491 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n82rn\" (UniqueName: \"kubernetes.io/projected/d508cbeb-a7cf-464f-88d4-b3374365e690-kube-api-access-n82rn\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:13.260864 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:13.260813 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-maas-api-tls\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:13.263510 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:13.263481 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-maas-api-tls\") pod \"maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:13.438592 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:13.438554 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:13.895914 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:13.895809 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5"] Nov 03 19:15:14.151695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:14.151622 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65005b3c-db12-4ddc-ab6e-2a4261ee6b69" path="/var/lib/kubelet/pods/65005b3c-db12-4ddc-ab6e-2a4261ee6b69/volumes" Nov 03 19:15:14.382919 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:14.382875 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" event={"ID":"d508cbeb-a7cf-464f-88d4-b3374365e690","Type":"ContainerStarted","Data":"36cda8e186eeb6d78a6517e766326dfd501700ccc8ba7828c313161d28a5904a"} Nov 03 19:15:15.075246 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.075213 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs"] Nov 03 19:15:15.076951 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.076894 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" Nov 03 19:15:15.077118 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.077104 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5455b012-cb7c-4ec5-bdc4-076bb20f9eae" containerName="istio-proxy" Nov 03 19:15:15.085692 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.084700 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.154975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.151445 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdn72\" (UniqueName: \"kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-kube-api-access-tdn72\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.154975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.151500 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-socket\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.154975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.151534 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641da692-25ae-4e05-9e06-2417ab73918d-istiod-ca-cert\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.154975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.151571 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-credential-socket\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.154975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.151596 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-istio-token\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.154975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.151660 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-envoy\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.154975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.151725 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-certs\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.154975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.151805 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-data\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.154975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.151973 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641da692-25ae-4e05-9e06-2417ab73918d-istio-podinfo\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.154975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.154759 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-9dcd6c-openshift-default-dockercfg-dqdsf\"" Nov 03 19:15:15.238356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.237772 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs"] Nov 03 19:15:15.246624 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.240036 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw"] Nov 03 19:15:15.246624 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.240265 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" containerID="cri-o://71590d9015078414b6fcffe0cce4392a3a3bc59a30f38cf60294d1903c54182f" gracePeriod=30 Nov 03 19:15:15.246624 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.246486 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh"] Nov 03 19:15:15.246981 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.246650 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" containerID="cri-o://06befcbed3c89285f103138870074aa0e5d2e1ebb33b7144c0025e7b2880f3de" gracePeriod=30 Nov 03 19:15:15.254864 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.250598 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn"] Nov 03 19:15:15.254864 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.250787 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" containerID="cri-o://b3f6d332491b29761a25e91e6b55c4e6cf62e9038914caaccca9cf1e5bc192f3" gracePeriod=30 Nov 03 19:15:15.254864 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.253133 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-certs\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.254864 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.253566 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-certs\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.254864 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.253739 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-data\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.254864 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.254035 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-data\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.263609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.255211 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641da692-25ae-4e05-9e06-2417ab73918d-istio-podinfo\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.263609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.255778 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdn72\" (UniqueName: \"kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-kube-api-access-tdn72\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.263609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.255808 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-socket\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.263609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.255840 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641da692-25ae-4e05-9e06-2417ab73918d-istiod-ca-cert\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.263609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.255881 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-credential-socket\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.263609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.255907 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-istio-token\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.263609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.255991 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-envoy\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.263609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.256839 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641da692-25ae-4e05-9e06-2417ab73918d-istiod-ca-cert\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.263609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.257057 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-socket\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.263609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.257292 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-credential-socket\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.267883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.267819 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641da692-25ae-4e05-9e06-2417ab73918d-istio-podinfo\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.270644 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.270589 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-envoy\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.273279 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.271648 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l"] Nov 03 19:15:15.273279 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.271955 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" containerID="cri-o://c8a8e39be54d67c70e4404ea82b40a05d54c16bdec0fc99751e4ea8ca7c341da" gracePeriod=30 Nov 03 19:15:15.314290 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.314253 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdn72\" (UniqueName: \"kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-kube-api-access-tdn72\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.315476 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.315452 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-istio-token\") pod \"e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.501363 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.479150 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:15.553067 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.543564 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" event={"ID":"d508cbeb-a7cf-464f-88d4-b3374365e690","Type":"ContainerStarted","Data":"0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b"} Nov 03 19:15:15.553067 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.544532 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:15.887939 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:15.887824 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" podStartSLOduration=2.904691764 podStartE2EDuration="3.887805793s" totalImagesPullingTime="983.114029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:15.881735428 +0000 UTC m=+1849.854296614" watchObservedRunningTime="2026-11-03 19:15:15.887805793 +0000 UTC m=+1849.860366977" Nov 03 19:15:16.050973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.042147 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs"] Nov 03 19:15:16.075884 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:15:16.075846 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641da692_25ae_4e05_9e06_2417ab73918d.slice/crio-b3f47afb84795f0bf20e525d797e7c9c37b79dc0fb69db09c470b006d3dab6fd WatchSource:0}: Error finding container b3f47afb84795f0bf20e525d797e7c9c37b79dc0fb69db09c470b006d3dab6fd: Status 404 returned error can't find the container with id b3f47afb84795f0bf20e525d797e7c9c37b79dc0fb69db09c470b006d3dab6fd Nov 03 19:15:16.081203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.079860 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:16.081203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.079964 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:16.081203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.080013 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:16.409204 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.409115 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl"] Nov 03 19:15:16.409407 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.409379 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" podUID="35207ff1-165f-4250-8315-7938cbf38842" containerName="istio-proxy" containerID="cri-o://b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1" gracePeriod=30 Nov 03 19:15:16.531371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.531295 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:15:16.531532 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.531394 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:15:16.553052 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.553003 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" event={"ID":"641da692-25ae-4e05-9e06-2417ab73918d","Type":"ContainerStarted","Data":"e59c1a0755e146175e8717f9766142ea60347220ffec2358f039f5b6c323445f"} Nov 03 19:15:16.553207 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.553073 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" event={"ID":"641da692-25ae-4e05-9e06-2417ab73918d","Type":"ContainerStarted","Data":"b3f47afb84795f0bf20e525d797e7c9c37b79dc0fb69db09c470b006d3dab6fd"} Nov 03 19:15:16.684419 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.684300 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" podStartSLOduration=2.6842751419999997 podStartE2EDuration="2.684275142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:16.666474527 +0000 UTC m=+1850.639035712" watchObservedRunningTime="2026-11-03 19:15:16.684275142 +0000 UTC m=+1850.656836321" Nov 03 19:15:16.778350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.775060 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:15:16.778350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.775134 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:15:16.835417 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.835378 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76"] Nov 03 19:15:16.835958 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.835921 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" containerID="cri-o://e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b" gracePeriod=30 Nov 03 19:15:16.939482 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.939387 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw"] Nov 03 19:15:16.939954 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.939906 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" containerID="cri-o://05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6" gracePeriod=30 Nov 03 19:15:16.966352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.966302 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d"] Nov 03 19:15:16.966817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.966783 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" containerID="cri-o://b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d" gracePeriod=30 Nov 03 19:15:16.976960 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.976925 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47"] Nov 03 19:15:16.977468 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:16.977394 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" podUID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" containerName="istio-proxy" containerID="cri-o://15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd" gracePeriod=30 Nov 03 19:15:17.490975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:17.490942 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:17.509818 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:17.509785 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:17.561561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:17.561529 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:17.562440 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:17.562415 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:15:17.942881 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:17.942483 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:15:17.942881 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:17.942543 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" podUID="35207ff1-165f-4250-8315-7938cbf38842" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:15:19.051058 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.051021 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl"] Nov 03 19:15:19.056307 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.056283 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.069538 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.069512 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk94r\" (UniqueName: \"kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-kube-api-access-jk94r\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.069673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.069563 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad088e1c-645f-4365-89ac-6e8f85589ddb-istiod-ca-cert\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.069743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.069696 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-credential-socket\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.069743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.069715 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-certs\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.069842 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.069818 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-data\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.069842 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.069836 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-envoy\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.069928 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.069902 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-podinfo\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.069992 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.069979 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-token\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.070039 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.070002 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-socket\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.105141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.105105 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4f1a90-openshift-default-dockercfg-tfvgg\"" Nov 03 19:15:19.219378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.214478 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-token\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.219378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.214528 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-socket\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.219378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.214614 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk94r\" (UniqueName: \"kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-kube-api-access-jk94r\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.219378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.214721 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad088e1c-645f-4365-89ac-6e8f85589ddb-istiod-ca-cert\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.219378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.214923 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-credential-socket\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.219378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.214957 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-certs\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.219378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.216520 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-credential-socket\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.219378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.216913 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-certs\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.238123 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.238052 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad088e1c-645f-4365-89ac-6e8f85589ddb-istiod-ca-cert\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.238523 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.238500 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-socket\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.238887 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.238868 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-data\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.282717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.282674 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-envoy\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.283563 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.239572 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-data\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.284698 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.283519 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-podinfo\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.403403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.331249 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk94r\" (UniqueName: \"kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-kube-api-access-jk94r\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.403403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.362305 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-token\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.403403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.363934 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl"] Nov 03 19:15:19.424163 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.420872 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-podinfo\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.440302 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.440268 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-envoy\") pod \"e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:19.441586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.441556 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:15:19.447716 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.447681 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" podUID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:15:19.676398 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:19.670770 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:20.381239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:20.372517 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:15:20.381239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:20.372571 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:15:20.618295 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:20.611781 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl"] Nov 03 19:15:20.633359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:20.630263 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:20.633359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:20.630380 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:20.633359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:20.630431 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:20.699354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:20.695516 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:15:20.699354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:20.695577 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 19:15:21.048353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.028640 5019 generic.go:356] "Generic (PLEG): container finished" podID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerID="c8a8e39be54d67c70e4404ea82b40a05d54c16bdec0fc99751e4ea8ca7c341da" exitCode=0 Nov 03 19:15:21.048353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.028836 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" event={"ID":"9d6f7402-2537-47a7-abc6-b21101126c4e","Type":"ContainerDied","Data":"c8a8e39be54d67c70e4404ea82b40a05d54c16bdec0fc99751e4ea8ca7c341da"} Nov 03 19:15:21.208008 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.207710 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:15:21.208008 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.207764 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:15:21.246560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.245933 5019 generic.go:356] "Generic (PLEG): container finished" podID="107848ae-227c-4b41-ab04-19b284e22fd3" containerID="06befcbed3c89285f103138870074aa0e5d2e1ebb33b7144c0025e7b2880f3de" exitCode=0 Nov 03 19:15:21.246560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.246028 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" event={"ID":"107848ae-227c-4b41-ab04-19b284e22fd3","Type":"ContainerDied","Data":"06befcbed3c89285f103138870074aa0e5d2e1ebb33b7144c0025e7b2880f3de"} Nov 03 19:15:21.246560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.246089 5019 scope.go:122] "RemoveContainer" containerID="a5780b7a2c8d2df3fc3eb7a6bf06d015194703070e816248c08274cc6c59c6fa" Nov 03 19:15:21.275552 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.275261 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:15:21.340995 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.340849 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" event={"ID":"ad088e1c-645f-4365-89ac-6e8f85589ddb","Type":"ContainerStarted","Data":"785cfcdc5c6bc68e4111118bfe6cf9da6809e4ec26eaf2984f8cbeafbe5726c3"} Nov 03 19:15:21.490851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.450613 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-token\" (UniqueName: \"kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-token\") pod \"85086af0-9f26-4517-88e8-8a3bd2219d2d\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " Nov 03 19:15:21.490851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.450689 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-data\") pod \"85086af0-9f26-4517-88e8-8a3bd2219d2d\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " Nov 03 19:15:21.490851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.450726 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-envoy\") pod \"85086af0-9f26-4517-88e8-8a3bd2219d2d\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " Nov 03 19:15:21.490851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.450774 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-socket\") pod \"85086af0-9f26-4517-88e8-8a3bd2219d2d\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " Nov 03 19:15:21.490851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.450898 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-podinfo\") pod \"85086af0-9f26-4517-88e8-8a3bd2219d2d\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " Nov 03 19:15:21.490851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.460479 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85086af0-9f26-4517-88e8-8a3bd2219d2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85086af0-9f26-4517-88e8-8a3bd2219d2d-istiod-ca-cert\") pod \"85086af0-9f26-4517-88e8-8a3bd2219d2d\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " Nov 03 19:15:21.490851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.460628 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-credential-socket\") pod \"85086af0-9f26-4517-88e8-8a3bd2219d2d\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " Nov 03 19:15:21.490851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.460771 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-certs\") pod \"85086af0-9f26-4517-88e8-8a3bd2219d2d\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.492435 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-kube-api-access-fbhsx\" (UniqueName: \"kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-kube-api-access-fbhsx\") pod \"85086af0-9f26-4517-88e8-8a3bd2219d2d\" (UID: \"85086af0-9f26-4517-88e8-8a3bd2219d2d\") " Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.495670 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-certs" pod "85086af0-9f26-4517-88e8-8a3bd2219d2d" (UID: "85086af0-9f26-4517-88e8-8a3bd2219d2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.500691 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85086af0-9f26-4517-88e8-8a3bd2219d2d-istiod-ca-cert" pod "85086af0-9f26-4517-88e8-8a3bd2219d2d" (UID: "85086af0-9f26-4517-88e8-8a3bd2219d2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.500939 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-credential-socket" pod "85086af0-9f26-4517-88e8-8a3bd2219d2d" (UID: "85086af0-9f26-4517-88e8-8a3bd2219d2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.501579 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-data" pod "85086af0-9f26-4517-88e8-8a3bd2219d2d" (UID: "85086af0-9f26-4517-88e8-8a3bd2219d2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.501744 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-socket" pod "85086af0-9f26-4517-88e8-8a3bd2219d2d" (UID: "85086af0-9f26-4517-88e8-8a3bd2219d2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.524800 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85086af0-9f26-4517-88e8-8a3bd2219d2d-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.524833 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.524849 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.524866 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.524881 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.525146 5019 generic.go:356] "Generic (PLEG): container finished" podID="48828f6f-de20-4e61-ad91-c648281f8304" containerID="b3f6d332491b29761a25e91e6b55c4e6cf62e9038914caaccca9cf1e5bc192f3" exitCode=0 Nov 03 19:15:21.537521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.525318 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" event={"ID":"48828f6f-de20-4e61-ad91-c648281f8304","Type":"ContainerDied","Data":"b3f6d332491b29761a25e91e6b55c4e6cf62e9038914caaccca9cf1e5bc192f3"} Nov 03 19:15:21.540733 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.539244 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-kube-api-access-fbhsx" pod "85086af0-9f26-4517-88e8-8a3bd2219d2d" (UID: "85086af0-9f26-4517-88e8-8a3bd2219d2d"). InnerVolumeSpecName "kube-api-access-fbhsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:21.542972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.542341 5019 generic.go:356] "Generic (PLEG): container finished" podID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerID="71590d9015078414b6fcffe0cce4392a3a3bc59a30f38cf60294d1903c54182f" exitCode=0 Nov 03 19:15:21.542972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.542380 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" event={"ID":"85086af0-9f26-4517-88e8-8a3bd2219d2d","Type":"ContainerDied","Data":"71590d9015078414b6fcffe0cce4392a3a3bc59a30f38cf60294d1903c54182f"} Nov 03 19:15:21.542972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.542486 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" Nov 03 19:15:21.639814 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.631910 5019 scope.go:122] "RemoveContainer" containerID="71590d9015078414b6fcffe0cce4392a3a3bc59a30f38cf60294d1903c54182f" Nov 03 19:15:21.669213 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.668586 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbhsx\" (UniqueName: \"kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-kube-api-access-fbhsx\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.678713 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.678665 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-podinfo" pod "85086af0-9f26-4517-88e8-8a3bd2219d2d" (UID: "85086af0-9f26-4517-88e8-8a3bd2219d2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:21.790181 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.786827 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.816695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.816640 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-envoy" pod "85086af0-9f26-4517-88e8-8a3bd2219d2d" (UID: "85086af0-9f26-4517-88e8-8a3bd2219d2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:21.889970 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.860432 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-token" pod "85086af0-9f26-4517-88e8-8a3bd2219d2d" (UID: "85086af0-9f26-4517-88e8-8a3bd2219d2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:21.892055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.890766 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.892055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.890800 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85086af0-9f26-4517-88e8-8a3bd2219d2d-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.939563 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.939194 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 19:15:21.995356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.995181 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-credential-socket\") pod \"107848ae-227c-4b41-ab04-19b284e22fd3\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " Nov 03 19:15:21.995356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.995255 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107848ae-227c-4b41-ab04-19b284e22fd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107848ae-227c-4b41-ab04-19b284e22fd3-istiod-ca-cert\") pod \"107848ae-227c-4b41-ab04-19b284e22fd3\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " Nov 03 19:15:21.995356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.995292 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-data\") pod \"107848ae-227c-4b41-ab04-19b284e22fd3\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " Nov 03 19:15:21.997355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.995322 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/107848ae-227c-4b41-ab04-19b284e22fd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107848ae-227c-4b41-ab04-19b284e22fd3-istio-podinfo\") pod \"107848ae-227c-4b41-ab04-19b284e22fd3\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " Nov 03 19:15:21.997355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.995673 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-socket\") pod \"107848ae-227c-4b41-ab04-19b284e22fd3\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " Nov 03 19:15:21.997355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.995771 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-istio-token\" (UniqueName: \"kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-istio-token\") pod \"107848ae-227c-4b41-ab04-19b284e22fd3\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " Nov 03 19:15:21.997355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.995837 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-kube-api-access-t9874\" (UniqueName: \"kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-kube-api-access-t9874\") pod \"107848ae-227c-4b41-ab04-19b284e22fd3\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " Nov 03 19:15:21.997355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.995999 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-envoy\") pod \"107848ae-227c-4b41-ab04-19b284e22fd3\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " Nov 03 19:15:21.997355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.996066 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-certs\") pod \"107848ae-227c-4b41-ab04-19b284e22fd3\" (UID: \"107848ae-227c-4b41-ab04-19b284e22fd3\") " Nov 03 19:15:21.998989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:21.998964 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-certs" pod "107848ae-227c-4b41-ab04-19b284e22fd3" (UID: "107848ae-227c-4b41-ab04-19b284e22fd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.000367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.000304 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-socket" pod "107848ae-227c-4b41-ab04-19b284e22fd3" (UID: "107848ae-227c-4b41-ab04-19b284e22fd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.001484 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.001454 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107848ae-227c-4b41-ab04-19b284e22fd3-istiod-ca-cert" pod "107848ae-227c-4b41-ab04-19b284e22fd3" (UID: "107848ae-227c-4b41-ab04-19b284e22fd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.001739 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.001715 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-data" pod "107848ae-227c-4b41-ab04-19b284e22fd3" (UID: "107848ae-227c-4b41-ab04-19b284e22fd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.004028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.003988 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-credential-socket" pod "107848ae-227c-4b41-ab04-19b284e22fd3" (UID: "107848ae-227c-4b41-ab04-19b284e22fd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.019167 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.019139 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:15:22.027895 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.027849 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-istio-token" pod "107848ae-227c-4b41-ab04-19b284e22fd3" (UID: "107848ae-227c-4b41-ab04-19b284e22fd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.028100 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.028072 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-envoy" pod "107848ae-227c-4b41-ab04-19b284e22fd3" (UID: "107848ae-227c-4b41-ab04-19b284e22fd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.028186 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.028152 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/107848ae-227c-4b41-ab04-19b284e22fd3-istio-podinfo" pod "107848ae-227c-4b41-ab04-19b284e22fd3" (UID: "107848ae-227c-4b41-ab04-19b284e22fd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:22.028453 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.028306 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-kube-api-access-t9874" pod "107848ae-227c-4b41-ab04-19b284e22fd3" (UID: "107848ae-227c-4b41-ab04-19b284e22fd3"). InnerVolumeSpecName "kube-api-access-t9874". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.110445 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-envoy\") pod \"9d6f7402-2537-47a7-abc6-b21101126c4e\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.110516 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d6f7402-2537-47a7-abc6-b21101126c4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6f7402-2537-47a7-abc6-b21101126c4e-istiod-ca-cert\") pod \"9d6f7402-2537-47a7-abc6-b21101126c4e\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.110652 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-data\") pod \"9d6f7402-2537-47a7-abc6-b21101126c4e\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.110682 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-token\" (UniqueName: \"kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-token\") pod \"9d6f7402-2537-47a7-abc6-b21101126c4e\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.110732 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-socket\") pod \"9d6f7402-2537-47a7-abc6-b21101126c4e\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.110761 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-podinfo\") pod \"9d6f7402-2537-47a7-abc6-b21101126c4e\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.110822 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-credential-socket\") pod \"9d6f7402-2537-47a7-abc6-b21101126c4e\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.110923 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-certs\") pod \"9d6f7402-2537-47a7-abc6-b21101126c4e\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.111017 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-kube-api-access-xljkc\" (UniqueName: \"kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-kube-api-access-xljkc\") pod \"9d6f7402-2537-47a7-abc6-b21101126c4e\" (UID: \"9d6f7402-2537-47a7-abc6-b21101126c4e\") " Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.111813 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.111838 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.111853 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.111868 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107848ae-227c-4b41-ab04-19b284e22fd3-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.111886 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.111901 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107848ae-227c-4b41-ab04-19b284e22fd3-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.111915 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107848ae-227c-4b41-ab04-19b284e22fd3-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.111928 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.111942 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9874\" (UniqueName: \"kubernetes.io/projected/107848ae-227c-4b41-ab04-19b284e22fd3-kube-api-access-t9874\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.116098 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-kube-api-access-xljkc" pod "9d6f7402-2537-47a7-abc6-b21101126c4e" (UID: "9d6f7402-2537-47a7-abc6-b21101126c4e"). InnerVolumeSpecName "kube-api-access-xljkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.119292 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-envoy" pod "9d6f7402-2537-47a7-abc6-b21101126c4e" (UID: "9d6f7402-2537-47a7-abc6-b21101126c4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.121669 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-podinfo" pod "9d6f7402-2537-47a7-abc6-b21101126c4e" (UID: "9d6f7402-2537-47a7-abc6-b21101126c4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.121872 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-credential-socket" pod "9d6f7402-2537-47a7-abc6-b21101126c4e" (UID: "9d6f7402-2537-47a7-abc6-b21101126c4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.122036 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-certs" pod "9d6f7402-2537-47a7-abc6-b21101126c4e" (UID: "9d6f7402-2537-47a7-abc6-b21101126c4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.122193 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-data" pod "9d6f7402-2537-47a7-abc6-b21101126c4e" (UID: "9d6f7402-2537-47a7-abc6-b21101126c4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.122464 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6f7402-2537-47a7-abc6-b21101126c4e-istiod-ca-cert" pod "9d6f7402-2537-47a7-abc6-b21101126c4e" (UID: "9d6f7402-2537-47a7-abc6-b21101126c4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.131595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.125441 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-socket" pod "9d6f7402-2537-47a7-abc6-b21101126c4e" (UID: "9d6f7402-2537-47a7-abc6-b21101126c4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.137982 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.134908 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-token" pod "9d6f7402-2537-47a7-abc6-b21101126c4e" (UID: "9d6f7402-2537-47a7-abc6-b21101126c4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.155755 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.154013 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:15:22.155755 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:15:22.153384 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc320ff_47d9_44dc_b85f_27848a66a7ec.slice/crio-15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fe1063_da3d_4283_a161_a9ce010d56fd.slice/crio-e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fe1063_da3d_4283_a161_a9ce010d56fd.slice/crio-conmon-e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d614c12_4321_4857_a576_9552dc037c70.slice/crio-05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6.scope\": RecentStats: unable to find data in memory cache]" Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.212318 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-kube-api-access-m97pw\" (UniqueName: \"kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-kube-api-access-m97pw\") pod \"48828f6f-de20-4e61-ad91-c648281f8304\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.212418 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-data\") pod \"48828f6f-de20-4e61-ad91-c648281f8304\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.212531 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48828f6f-de20-4e61-ad91-c648281f8304-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48828f6f-de20-4e61-ad91-c648281f8304-istiod-ca-cert\") pod \"48828f6f-de20-4e61-ad91-c648281f8304\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.212565 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-socket\") pod \"48828f6f-de20-4e61-ad91-c648281f8304\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.212627 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48828f6f-de20-4e61-ad91-c648281f8304-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48828f6f-de20-4e61-ad91-c648281f8304-istio-podinfo\") pod \"48828f6f-de20-4e61-ad91-c648281f8304\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.212703 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-istio-token\" (UniqueName: \"kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-istio-token\") pod \"48828f6f-de20-4e61-ad91-c648281f8304\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.212799 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-credential-socket\") pod \"48828f6f-de20-4e61-ad91-c648281f8304\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.212840 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-certs\") pod \"48828f6f-de20-4e61-ad91-c648281f8304\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.212917 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-envoy\") pod \"48828f6f-de20-4e61-ad91-c648281f8304\" (UID: \"48828f6f-de20-4e61-ad91-c648281f8304\") " Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.213756 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xljkc\" (UniqueName: \"kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-kube-api-access-xljkc\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.213782 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.213796 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6f7402-2537-47a7-abc6-b21101126c4e-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.213810 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.213823 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.213837 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.213850 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6f7402-2537-47a7-abc6-b21101126c4e-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.213864 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.213877 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6f7402-2537-47a7-abc6-b21101126c4e-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.217457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.215511 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-certs" pod "48828f6f-de20-4e61-ad91-c648281f8304" (UID: "48828f6f-de20-4e61-ad91-c648281f8304"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.234044 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.234006 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-data" pod "48828f6f-de20-4e61-ad91-c648281f8304" (UID: "48828f6f-de20-4e61-ad91-c648281f8304"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.234347 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.234310 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48828f6f-de20-4e61-ad91-c648281f8304-istiod-ca-cert" pod "48828f6f-de20-4e61-ad91-c648281f8304" (UID: "48828f6f-de20-4e61-ad91-c648281f8304"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.236289 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.236258 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-envoy" pod "48828f6f-de20-4e61-ad91-c648281f8304" (UID: "48828f6f-de20-4e61-ad91-c648281f8304"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.237050 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.237012 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-socket" pod "48828f6f-de20-4e61-ad91-c648281f8304" (UID: "48828f6f-de20-4e61-ad91-c648281f8304"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.247090 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.237561 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48828f6f-de20-4e61-ad91-c648281f8304-istio-podinfo" pod "48828f6f-de20-4e61-ad91-c648281f8304" (UID: "48828f6f-de20-4e61-ad91-c648281f8304"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:22.247090 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.239193 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-credential-socket" pod "48828f6f-de20-4e61-ad91-c648281f8304" (UID: "48828f6f-de20-4e61-ad91-c648281f8304"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.257046 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.255908 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-istio-token" pod "48828f6f-de20-4e61-ad91-c648281f8304" (UID: "48828f6f-de20-4e61-ad91-c648281f8304"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.269228 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.269180 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-kube-api-access-m97pw" pod "48828f6f-de20-4e61-ad91-c648281f8304" (UID: "48828f6f-de20-4e61-ad91-c648281f8304"). InnerVolumeSpecName "kube-api-access-m97pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.270159 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.270134 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:15:22.314947 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.314911 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-envoy\") pod \"35207ff1-165f-4250-8315-7938cbf38842\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " Nov 03 19:15:22.315148 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.315061 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-kube-api-access-rsqdt\" (UniqueName: \"kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-kube-api-access-rsqdt\") pod \"35207ff1-165f-4250-8315-7938cbf38842\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " Nov 03 19:15:22.315148 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.315134 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-data\") pod \"35207ff1-165f-4250-8315-7938cbf38842\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " Nov 03 19:15:22.315885 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.315858 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35207ff1-165f-4250-8315-7938cbf38842-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35207ff1-165f-4250-8315-7938cbf38842-istio-podinfo\") pod \"35207ff1-165f-4250-8315-7938cbf38842\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " Nov 03 19:15:22.315989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.315971 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-socket\") pod \"35207ff1-165f-4250-8315-7938cbf38842\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " Nov 03 19:15:22.316061 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.316023 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-certs\") pod \"35207ff1-165f-4250-8315-7938cbf38842\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " Nov 03 19:15:22.316160 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.316118 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-credential-socket\") pod \"35207ff1-165f-4250-8315-7938cbf38842\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " Nov 03 19:15:22.316160 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.316150 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35207ff1-165f-4250-8315-7938cbf38842-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35207ff1-165f-4250-8315-7938cbf38842-istiod-ca-cert\") pod \"35207ff1-165f-4250-8315-7938cbf38842\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " Nov 03 19:15:22.316278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.316230 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-istio-token\" (UniqueName: \"kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-istio-token\") pod \"35207ff1-165f-4250-8315-7938cbf38842\" (UID: \"35207ff1-165f-4250-8315-7938cbf38842\") " Nov 03 19:15:22.317519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.316714 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-certs" pod "35207ff1-165f-4250-8315-7938cbf38842" (UID: "35207ff1-165f-4250-8315-7938cbf38842"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.317519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.317073 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m97pw\" (UniqueName: \"kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-kube-api-access-m97pw\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.317519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.317099 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.317519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.317115 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48828f6f-de20-4e61-ad91-c648281f8304-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.317519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.317129 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.317519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.317142 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48828f6f-de20-4e61-ad91-c648281f8304-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.317519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.317156 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48828f6f-de20-4e61-ad91-c648281f8304-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.317519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.317170 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.317519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.317184 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.317519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.317198 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.317519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.317212 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48828f6f-de20-4e61-ad91-c648281f8304-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.318935 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.318883 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35207ff1-165f-4250-8315-7938cbf38842-istiod-ca-cert" pod "35207ff1-165f-4250-8315-7938cbf38842" (UID: "35207ff1-165f-4250-8315-7938cbf38842"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.319147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.319076 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-credential-socket" pod "35207ff1-165f-4250-8315-7938cbf38842" (UID: "35207ff1-165f-4250-8315-7938cbf38842"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.319835 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.319794 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-socket" pod "35207ff1-165f-4250-8315-7938cbf38842" (UID: "35207ff1-165f-4250-8315-7938cbf38842"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.320086 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.320056 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-istio-token" pod "35207ff1-165f-4250-8315-7938cbf38842" (UID: "35207ff1-165f-4250-8315-7938cbf38842"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.320453 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.320422 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-envoy" pod "35207ff1-165f-4250-8315-7938cbf38842" (UID: "35207ff1-165f-4250-8315-7938cbf38842"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.320595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.320578 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-data" pod "35207ff1-165f-4250-8315-7938cbf38842" (UID: "35207ff1-165f-4250-8315-7938cbf38842"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.321146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.321083 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-kube-api-access-rsqdt" pod "35207ff1-165f-4250-8315-7938cbf38842" (UID: "35207ff1-165f-4250-8315-7938cbf38842"). InnerVolumeSpecName "kube-api-access-rsqdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.321146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.321112 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35207ff1-165f-4250-8315-7938cbf38842-istio-podinfo" pod "35207ff1-165f-4250-8315-7938cbf38842" (UID: "35207ff1-165f-4250-8315-7938cbf38842"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:22.419270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.419120 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.419270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.419160 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35207ff1-165f-4250-8315-7938cbf38842-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.419270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.419174 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.419270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.419189 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.419270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.419203 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsqdt\" (UniqueName: \"kubernetes.io/projected/35207ff1-165f-4250-8315-7938cbf38842-kube-api-access-rsqdt\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.419270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.419218 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.419270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.419231 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35207ff1-165f-4250-8315-7938cbf38842-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.419270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.419244 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35207ff1-165f-4250-8315-7938cbf38842-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.538886 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.538865 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:15:22.543506 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.543477 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:15:22.557393 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.557304 5019 generic.go:356] "Generic (PLEG): container finished" podID="35207ff1-165f-4250-8315-7938cbf38842" containerID="b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1" exitCode=0 Nov 03 19:15:22.557527 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.557418 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" event={"ID":"35207ff1-165f-4250-8315-7938cbf38842","Type":"ContainerDied","Data":"b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1"} Nov 03 19:15:22.557527 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.557455 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" event={"ID":"35207ff1-165f-4250-8315-7938cbf38842","Type":"ContainerDied","Data":"77abc2b593b30697cd3e714bf32305d56f4fe9acd66bbf8bf3d0c210f390bdb5"} Nov 03 19:15:22.557527 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.557477 5019 scope.go:122] "RemoveContainer" containerID="b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1" Nov 03 19:15:22.557696 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.557570 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl" Nov 03 19:15:22.561104 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.560860 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw"] Nov 03 19:15:22.561166 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.561105 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw" event={"ID":"85086af0-9f26-4517-88e8-8a3bd2219d2d","Type":"ContainerDied","Data":"4fcb94f732eb91acc7c1188c8232839e35bda3a9406b9fe05d520ea8d5c3980d"} Nov 03 19:15:22.566145 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.562941 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:15:22.566145 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.563432 5019 generic.go:356] "Generic (PLEG): container finished" podID="4d614c12-4321-4857-a576-9552dc037c70" containerID="05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6" exitCode=0 Nov 03 19:15:22.566145 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.563787 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" event={"ID":"4d614c12-4321-4857-a576-9552dc037c70","Type":"ContainerDied","Data":"05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6"} Nov 03 19:15:22.566145 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.563835 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" event={"ID":"4d614c12-4321-4857-a576-9552dc037c70","Type":"ContainerDied","Data":"6db90f28d18f3d55c7489fb6749a2e1d4a982ccfb2eb12302883b082dc180cbf"} Nov 03 19:15:22.568240 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.568130 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" event={"ID":"9d6f7402-2537-47a7-abc6-b21101126c4e","Type":"ContainerDied","Data":"13a65c293f5d6fff40466b6091e7f99c052e50098fb5385b4ccb9d350da12069"} Nov 03 19:15:22.568240 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.568165 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l" Nov 03 19:15:22.571234 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.571097 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:15:22.572620 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.571817 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:15:22.572705 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.572683 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-w55tw"] Nov 03 19:15:22.586931 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.586521 5019 generic.go:356] "Generic (PLEG): container finished" podID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerID="b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d" exitCode=0 Nov 03 19:15:22.586931 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.586658 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" event={"ID":"e039bad7-2408-4ef8-8d43-4e3a540a6c56","Type":"ContainerDied","Data":"b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d"} Nov 03 19:15:22.586931 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.586700 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" event={"ID":"e039bad7-2408-4ef8-8d43-4e3a540a6c56","Type":"ContainerDied","Data":"7257f0f987959b85fc5ea1a9f7ccc43c4f27519db51475b3bde7e6fb902f6aa5"} Nov 03 19:15:22.590274 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.590252 5019 generic.go:356] "Generic (PLEG): container finished" podID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" containerID="15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd" exitCode=0 Nov 03 19:15:22.590448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.590279 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" event={"ID":"9fc320ff-47d9-44dc-b85f-27848a66a7ec","Type":"ContainerDied","Data":"15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd"} Nov 03 19:15:22.590448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.590298 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" Nov 03 19:15:22.590448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.590387 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47" event={"ID":"9fc320ff-47d9-44dc-b85f-27848a66a7ec","Type":"ContainerDied","Data":"aedbff0eb92efba4293fdbe05765429fc341a1ffd0f5bcacf47827672019bc5c"} Nov 03 19:15:22.592180 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.592162 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" Nov 03 19:15:22.592349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.592176 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh" event={"ID":"107848ae-227c-4b41-ab04-19b284e22fd3","Type":"ContainerDied","Data":"d0e346b6b27d1e4a9fe06ed0a3723fc625c530145ff206956986bc3d03d45974"} Nov 03 19:15:22.594418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.594394 5019 generic.go:356] "Generic (PLEG): container finished" podID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerID="e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b" exitCode=0 Nov 03 19:15:22.594495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.594468 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" event={"ID":"b1fe1063-da3d-4283-a161-a9ce010d56fd","Type":"ContainerDied","Data":"e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b"} Nov 03 19:15:22.594568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.594508 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" event={"ID":"b1fe1063-da3d-4283-a161-a9ce010d56fd","Type":"ContainerDied","Data":"ed705e35160ea980135615f29a2c0013e9c82f890c336c5705672b053025f5dc"} Nov 03 19:15:22.594619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.594575 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76" Nov 03 19:15:22.596747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.596693 5019 scope.go:122] "RemoveContainer" containerID="b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1" Nov 03 19:15:22.596995 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:15:22.596972 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1\": container with ID starting with b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1 not found: ID does not exist" containerID="b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1" Nov 03 19:15:22.597073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.597006 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1"} err="failed to get container status \"b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1\": rpc error: code = NotFound desc = could not find container \"b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1\": container with ID starting with b8b5f0a2e5d55bffa4f03b7c14fb83478049931fcf0820702f34cd7dd25431e1 not found: ID does not exist" Nov 03 19:15:22.597073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.597030 5019 scope.go:122] "RemoveContainer" containerID="05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6" Nov 03 19:15:22.599520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.598854 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" event={"ID":"ad088e1c-645f-4365-89ac-6e8f85589ddb","Type":"ContainerStarted","Data":"3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de"} Nov 03 19:15:22.601317 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.601294 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" event={"ID":"48828f6f-de20-4e61-ad91-c648281f8304","Type":"ContainerDied","Data":"98201635882f8cb20be94936818cac753b259c3d46e600df1aba83c233a040e1"} Nov 03 19:15:22.601462 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.601353 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn" Nov 03 19:15:22.612522 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.612506 5019 scope.go:122] "RemoveContainer" containerID="05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6" Nov 03 19:15:22.612773 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:15:22.612751 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6\": container with ID starting with 05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6 not found: ID does not exist" containerID="05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6" Nov 03 19:15:22.612875 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.612778 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6"} err="failed to get container status \"05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6\": rpc error: code = NotFound desc = could not find container \"05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6\": container with ID starting with 05c7a25130b0a602829f6c9b45f42a3c23c43e9dd6ab0912ef690cd86ca14ec6 not found: ID does not exist" Nov 03 19:15:22.612875 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.612796 5019 scope.go:122] "RemoveContainer" containerID="c8a8e39be54d67c70e4404ea82b40a05d54c16bdec0fc99751e4ea8ca7c341da" Nov 03 19:15:22.621700 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.621660 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-socket\") pod \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " Nov 03 19:15:22.621815 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.621702 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-credential-socket\") pod \"b1fe1063-da3d-4283-a161-a9ce010d56fd\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " Nov 03 19:15:22.621815 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.621760 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-socket\") pod \"4d614c12-4321-4857-a576-9552dc037c70\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " Nov 03 19:15:22.621815 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.621792 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-podinfo\") pod \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " Nov 03 19:15:22.622059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.621849 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d614c12-4321-4857-a576-9552dc037c70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d614c12-4321-4857-a576-9552dc037c70-istiod-ca-cert\") pod \"4d614c12-4321-4857-a576-9552dc037c70\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " Nov 03 19:15:22.622059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.621881 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-istio-token\" (UniqueName: \"kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-istio-token\") pod \"4d614c12-4321-4857-a576-9552dc037c70\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " Nov 03 19:15:22.622059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.621926 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-certs\") pod \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " Nov 03 19:15:22.622059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.621963 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-envoy\") pod \"4d614c12-4321-4857-a576-9552dc037c70\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " Nov 03 19:15:22.622059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622003 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-data\") pod \"4d614c12-4321-4857-a576-9552dc037c70\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " Nov 03 19:15:22.622059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622031 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istiod-ca-cert\") pod \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " Nov 03 19:15:22.622486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622065 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-data\") pod \"b1fe1063-da3d-4283-a161-a9ce010d56fd\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " Nov 03 19:15:22.622486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622117 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-token\" (UniqueName: \"kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-token\") pod \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " Nov 03 19:15:22.622486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622145 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-socket\") pod \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " Nov 03 19:15:22.622486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622180 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-kube-api-access-l5vps\" (UniqueName: \"kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-kube-api-access-l5vps\") pod \"4d614c12-4321-4857-a576-9552dc037c70\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " Nov 03 19:15:22.622486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622270 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-data\") pod \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " Nov 03 19:15:22.622486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622313 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-envoy\") pod \"b1fe1063-da3d-4283-a161-a9ce010d56fd\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " Nov 03 19:15:22.622486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622417 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-certs\") pod \"4d614c12-4321-4857-a576-9552dc037c70\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " Nov 03 19:15:22.622486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622458 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-certs\") pod \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " Nov 03 19:15:22.622989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622488 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-podinfo\") pod \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " Nov 03 19:15:22.622989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622526 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-certs\") pod \"b1fe1063-da3d-4283-a161-a9ce010d56fd\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " Nov 03 19:15:22.622989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622576 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-envoy\") pod \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " Nov 03 19:15:22.622989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622644 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-credential-socket\") pod \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " Nov 03 19:15:22.622989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622691 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-credential-socket\") pod \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " Nov 03 19:15:22.622989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622753 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-envoy\") pod \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " Nov 03 19:15:22.622989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622798 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-token\" (UniqueName: \"kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-token\") pod \"b1fe1063-da3d-4283-a161-a9ce010d56fd\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " Nov 03 19:15:22.622989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622851 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-data\") pod \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " Nov 03 19:15:22.622989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622900 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-kube-api-access-9tvw6\" (UniqueName: \"kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-kube-api-access-9tvw6\") pod \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\" (UID: \"e039bad7-2408-4ef8-8d43-4e3a540a6c56\") " Nov 03 19:15:22.622989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.622945 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-kube-api-access-c5g5p\" (UniqueName: \"kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-kube-api-access-c5g5p\") pod \"b1fe1063-da3d-4283-a161-a9ce010d56fd\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " Nov 03 19:15:22.623636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.623015 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-token\" (UniqueName: \"kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-token\") pod \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " Nov 03 19:15:22.623636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.623082 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1fe1063-da3d-4283-a161-a9ce010d56fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fe1063-da3d-4283-a161-a9ce010d56fd-istiod-ca-cert\") pod \"b1fe1063-da3d-4283-a161-a9ce010d56fd\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " Nov 03 19:15:22.623636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.623123 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-podinfo\") pod \"b1fe1063-da3d-4283-a161-a9ce010d56fd\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " Nov 03 19:15:22.623636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.623170 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istiod-ca-cert\") pod \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " Nov 03 19:15:22.623636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.623199 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-credential-socket\") pod \"4d614c12-4321-4857-a576-9552dc037c70\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " Nov 03 19:15:22.623636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.623239 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d614c12-4321-4857-a576-9552dc037c70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d614c12-4321-4857-a576-9552dc037c70-istio-podinfo\") pod \"4d614c12-4321-4857-a576-9552dc037c70\" (UID: \"4d614c12-4321-4857-a576-9552dc037c70\") " Nov 03 19:15:22.623636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.623270 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-socket\") pod \"b1fe1063-da3d-4283-a161-a9ce010d56fd\" (UID: \"b1fe1063-da3d-4283-a161-a9ce010d56fd\") " Nov 03 19:15:22.623636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.623302 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-kube-api-access-lbcxg\" (UniqueName: \"kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-kube-api-access-lbcxg\") pod \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\" (UID: \"9fc320ff-47d9-44dc-b85f-27848a66a7ec\") " Nov 03 19:15:22.624589 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.624342 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-credential-socket" pod "9fc320ff-47d9-44dc-b85f-27848a66a7ec" (UID: "9fc320ff-47d9-44dc-b85f-27848a66a7ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.625036 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.625016 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istiod-ca-cert" pod "9fc320ff-47d9-44dc-b85f-27848a66a7ec" (UID: "9fc320ff-47d9-44dc-b85f-27848a66a7ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.625317 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.625215 5019 scope.go:122] "RemoveContainer" containerID="b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d" Nov 03 19:15:22.625562 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.625543 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fe1063-da3d-4283-a161-a9ce010d56fd-istiod-ca-cert" pod "b1fe1063-da3d-4283-a161-a9ce010d56fd" (UID: "b1fe1063-da3d-4283-a161-a9ce010d56fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.625646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.625552 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-certs" pod "b1fe1063-da3d-4283-a161-a9ce010d56fd" (UID: "b1fe1063-da3d-4283-a161-a9ce010d56fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.626177 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.626083 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-data" pod "4d614c12-4321-4857-a576-9552dc037c70" (UID: "4d614c12-4321-4857-a576-9552dc037c70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.626283 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.626234 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-credential-socket" pod "4d614c12-4321-4857-a576-9552dc037c70" (UID: "4d614c12-4321-4857-a576-9552dc037c70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.626366 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.626300 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-socket" pod "9fc320ff-47d9-44dc-b85f-27848a66a7ec" (UID: "9fc320ff-47d9-44dc-b85f-27848a66a7ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.626492 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.626471 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-certs" pod "9fc320ff-47d9-44dc-b85f-27848a66a7ec" (UID: "9fc320ff-47d9-44dc-b85f-27848a66a7ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.626831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.626677 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-socket" pod "4d614c12-4321-4857-a576-9552dc037c70" (UID: "4d614c12-4321-4857-a576-9552dc037c70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.627364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.627343 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-data" pod "b1fe1063-da3d-4283-a161-a9ce010d56fd" (UID: "b1fe1063-da3d-4283-a161-a9ce010d56fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.627698 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.627669 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d614c12-4321-4857-a576-9552dc037c70-istiod-ca-cert" pod "4d614c12-4321-4857-a576-9552dc037c70" (UID: "4d614c12-4321-4857-a576-9552dc037c70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.628378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.628307 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-credential-socket" pod "e039bad7-2408-4ef8-8d43-4e3a540a6c56" (UID: "e039bad7-2408-4ef8-8d43-4e3a540a6c56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.629382 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.629307 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-data" pod "e039bad7-2408-4ef8-8d43-4e3a540a6c56" (UID: "e039bad7-2408-4ef8-8d43-4e3a540a6c56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.630507 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.630413 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istiod-ca-cert" pod "e039bad7-2408-4ef8-8d43-4e3a540a6c56" (UID: "e039bad7-2408-4ef8-8d43-4e3a540a6c56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.630507 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.630468 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-socket" pod "e039bad7-2408-4ef8-8d43-4e3a540a6c56" (UID: "e039bad7-2408-4ef8-8d43-4e3a540a6c56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.631594 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.631566 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-kube-api-access-l5vps" pod "4d614c12-4321-4857-a576-9552dc037c70" (UID: "4d614c12-4321-4857-a576-9552dc037c70"). InnerVolumeSpecName "kube-api-access-l5vps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.631711 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.631629 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-podinfo" pod "e039bad7-2408-4ef8-8d43-4e3a540a6c56" (UID: "e039bad7-2408-4ef8-8d43-4e3a540a6c56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:22.633131 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.633087 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-certs" pod "e039bad7-2408-4ef8-8d43-4e3a540a6c56" (UID: "e039bad7-2408-4ef8-8d43-4e3a540a6c56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.634073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.633928 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-data" pod "9fc320ff-47d9-44dc-b85f-27848a66a7ec" (UID: "9fc320ff-47d9-44dc-b85f-27848a66a7ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.635208 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.626515 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-credential-socket" pod "b1fe1063-da3d-4283-a161-a9ce010d56fd" (UID: "b1fe1063-da3d-4283-a161-a9ce010d56fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.635208 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.634411 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-certs" pod "4d614c12-4321-4857-a576-9552dc037c70" (UID: "4d614c12-4321-4857-a576-9552dc037c70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.635208 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.634886 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-socket" pod "b1fe1063-da3d-4283-a161-a9ce010d56fd" (UID: "b1fe1063-da3d-4283-a161-a9ce010d56fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.639280 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.639253 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-envoy" pod "4d614c12-4321-4857-a576-9552dc037c70" (UID: "4d614c12-4321-4857-a576-9552dc037c70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.640495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.640470 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-token" pod "b1fe1063-da3d-4283-a161-a9ce010d56fd" (UID: "b1fe1063-da3d-4283-a161-a9ce010d56fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.640603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.640499 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-token" pod "e039bad7-2408-4ef8-8d43-4e3a540a6c56" (UID: "e039bad7-2408-4ef8-8d43-4e3a540a6c56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.640603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.640511 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-istio-token" pod "4d614c12-4321-4857-a576-9552dc037c70" (UID: "4d614c12-4321-4857-a576-9552dc037c70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.640603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.640530 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-envoy" pod "9fc320ff-47d9-44dc-b85f-27848a66a7ec" (UID: "9fc320ff-47d9-44dc-b85f-27848a66a7ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.640603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.640560 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-envoy" pod "e039bad7-2408-4ef8-8d43-4e3a540a6c56" (UID: "e039bad7-2408-4ef8-8d43-4e3a540a6c56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.640805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.640639 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-kube-api-access-c5g5p" pod "b1fe1063-da3d-4283-a161-a9ce010d56fd" (UID: "b1fe1063-da3d-4283-a161-a9ce010d56fd"). InnerVolumeSpecName "kube-api-access-c5g5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.640805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.640638 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-podinfo" pod "b1fe1063-da3d-4283-a161-a9ce010d56fd" (UID: "b1fe1063-da3d-4283-a161-a9ce010d56fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:22.640805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.640721 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-podinfo" pod "9fc320ff-47d9-44dc-b85f-27848a66a7ec" (UID: "9fc320ff-47d9-44dc-b85f-27848a66a7ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:22.640805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.640710 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d614c12-4321-4857-a576-9552dc037c70-istio-podinfo" pod "4d614c12-4321-4857-a576-9552dc037c70" (UID: "4d614c12-4321-4857-a576-9552dc037c70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:22.640805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.640769 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-token" pod "9fc320ff-47d9-44dc-b85f-27848a66a7ec" (UID: "9fc320ff-47d9-44dc-b85f-27848a66a7ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.640805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.640793 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-kube-api-access-9tvw6" pod "e039bad7-2408-4ef8-8d43-4e3a540a6c56" (UID: "e039bad7-2408-4ef8-8d43-4e3a540a6c56"). InnerVolumeSpecName "kube-api-access-9tvw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.640991 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.640851 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-kube-api-access-lbcxg" pod "9fc320ff-47d9-44dc-b85f-27848a66a7ec" (UID: "9fc320ff-47d9-44dc-b85f-27848a66a7ec"). InnerVolumeSpecName "kube-api-access-lbcxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.642111 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.642090 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-envoy" pod "b1fe1063-da3d-4283-a161-a9ce010d56fd" (UID: "b1fe1063-da3d-4283-a161-a9ce010d56fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:22.646248 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.646221 5019 scope.go:122] "RemoveContainer" containerID="b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d" Nov 03 19:15:22.646617 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:15:22.646594 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d\": container with ID starting with b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d not found: ID does not exist" containerID="b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d" Nov 03 19:15:22.646704 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.646624 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d"} err="failed to get container status \"b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d\": rpc error: code = NotFound desc = could not find container \"b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d\": container with ID starting with b53fdb15b157ab8d815fb33cca8df39760907b736ee40679dce725274de3de8d not found: ID does not exist" Nov 03 19:15:22.646704 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.646641 5019 scope.go:122] "RemoveContainer" containerID="15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd" Nov 03 19:15:22.658028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.658010 5019 scope.go:122] "RemoveContainer" containerID="15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd" Nov 03 19:15:22.658400 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:15:22.658375 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd\": container with ID starting with 15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd not found: ID does not exist" containerID="15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd" Nov 03 19:15:22.658473 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.658414 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd"} err="failed to get container status \"15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd\": rpc error: code = NotFound desc = could not find container \"15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd\": container with ID starting with 15d06afe39d61708dab46bbe4206257d1854db194968eab9ecc04e1eab5c2bfd not found: ID does not exist" Nov 03 19:15:22.658473 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.658439 5019 scope.go:122] "RemoveContainer" containerID="06befcbed3c89285f103138870074aa0e5d2e1ebb33b7144c0025e7b2880f3de" Nov 03 19:15:22.668643 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.668621 5019 scope.go:122] "RemoveContainer" containerID="e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b" Nov 03 19:15:22.673901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.673876 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:22.682020 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.681993 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:22.682440 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.682396 5019 scope.go:122] "RemoveContainer" containerID="e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b" Nov 03 19:15:22.682804 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:15:22.682775 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b\": container with ID starting with e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b not found: ID does not exist" containerID="e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b" Nov 03 19:15:22.682891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.682808 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b"} err="failed to get container status \"e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b\": rpc error: code = NotFound desc = could not find container \"e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b\": container with ID starting with e0fad128f0d63a69f74a405607ee75adb80cba4daf23e0baeb2454df0673b78b not found: ID does not exist" Nov 03 19:15:22.682891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.682832 5019 scope.go:122] "RemoveContainer" containerID="b3f6d332491b29761a25e91e6b55c4e6cf62e9038914caaccca9cf1e5bc192f3" Nov 03 19:15:22.725913 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.725880 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.725913 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.725907 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.725925 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.725939 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.725953 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.725966 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.725979 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.725992 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726006 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726019 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726033 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tvw6\" (UniqueName: \"kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-kube-api-access-9tvw6\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726047 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5g5p\" (UniqueName: \"kubernetes.io/projected/b1fe1063-da3d-4283-a161-a9ce010d56fd-kube-api-access-c5g5p\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726062 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726075 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fe1063-da3d-4283-a161-a9ce010d56fd-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726089 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726104 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc320ff-47d9-44dc-b85f-27848a66a7ec-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726119 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726135 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d614c12-4321-4857-a576-9552dc037c70-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726150 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726164 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbcxg\" (UniqueName: \"kubernetes.io/projected/9fc320ff-47d9-44dc-b85f-27848a66a7ec-kube-api-access-lbcxg\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726174 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc320ff-47d9-44dc-b85f-27848a66a7ec-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726182 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726190 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726200 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726208 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d614c12-4321-4857-a576-9552dc037c70-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726217 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726226 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726234 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726241 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d614c12-4321-4857-a576-9552dc037c70-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726252 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726260 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726268 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726276 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726285 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5vps\" (UniqueName: \"kubernetes.io/projected/4d614c12-4321-4857-a576-9552dc037c70-kube-api-access-l5vps\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726293 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e039bad7-2408-4ef8-8d43-4e3a540a6c56-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.726827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:22.726301 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fe1063-da3d-4283-a161-a9ce010d56fd-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:23.009502 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.009399 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn"] Nov 03 19:15:23.014902 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.014873 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-cdgjn"] Nov 03 19:15:23.125606 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.125569 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl"] Nov 03 19:15:23.128988 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.128960 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-nqdfl"] Nov 03 19:15:23.228149 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.228109 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh"] Nov 03 19:15:23.231987 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.231957 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-s7srh"] Nov 03 19:15:23.497800 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.497758 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l"] Nov 03 19:15:23.502797 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.502770 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-lkt7l"] Nov 03 19:15:23.610242 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.610127 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" podStartSLOduration=4.610115072 podStartE2EDuration="4.610115072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:23.60893677 +0000 UTC m=+1857.581497952" watchObservedRunningTime="2026-11-03 19:15:23.610115072 +0000 UTC m=+1857.582676254" Nov 03 19:15:23.616068 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.616045 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw" Nov 03 19:15:23.619201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.619170 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d" Nov 03 19:15:23.623127 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.623105 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:23.624277 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.624244 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:15:23.978487 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.978453 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d"] Nov 03 19:15:23.987534 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:23.987501 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-t4d4d"] Nov 03 19:15:24.049701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.049666 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76"] Nov 03 19:15:24.054019 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.053989 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-shp76"] Nov 03 19:15:24.154633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.154598 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" path="/var/lib/kubelet/pods/107848ae-227c-4b41-ab04-19b284e22fd3/volumes" Nov 03 19:15:24.155136 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.155120 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35207ff1-165f-4250-8315-7938cbf38842" path="/var/lib/kubelet/pods/35207ff1-165f-4250-8315-7938cbf38842/volumes" Nov 03 19:15:24.155722 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.155698 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48828f6f-de20-4e61-ad91-c648281f8304" path="/var/lib/kubelet/pods/48828f6f-de20-4e61-ad91-c648281f8304/volumes" Nov 03 19:15:24.156659 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.156642 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" path="/var/lib/kubelet/pods/85086af0-9f26-4517-88e8-8a3bd2219d2d/volumes" Nov 03 19:15:24.157135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.157119 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" path="/var/lib/kubelet/pods/9d6f7402-2537-47a7-abc6-b21101126c4e/volumes" Nov 03 19:15:24.157595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.157582 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" path="/var/lib/kubelet/pods/b1fe1063-da3d-4283-a161-a9ce010d56fd/volumes" Nov 03 19:15:24.158371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.158354 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" path="/var/lib/kubelet/pods/e039bad7-2408-4ef8-8d43-4e3a540a6c56/volumes" Nov 03 19:15:24.158692 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.158678 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw"] Nov 03 19:15:24.162356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.162306 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-pxctw"] Nov 03 19:15:24.265415 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.265310 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47"] Nov 03 19:15:24.410612 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.406124 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-45h47"] Nov 03 19:15:24.592216 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.592129 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4"] Nov 03 19:15:24.593362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593316 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" Nov 03 19:15:24.593362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593351 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593375 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593384 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593398 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593403 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593416 5019 cpu_manager.go:414] "containerMap: removing container" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593422 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593431 5019 cpu_manager.go:414] "containerMap: removing container" podUID="35207ff1-165f-4250-8315-7938cbf38842" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593436 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35207ff1-165f-4250-8315-7938cbf38842" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593444 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593448 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593460 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593464 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593472 5019 cpu_manager.go:414] "containerMap: removing container" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593476 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593487 5019 cpu_manager.go:414] "containerMap: removing container" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593491 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" Nov 03 19:15:24.593505 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593513 5019 cpu_manager.go:414] "containerMap: removing container" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" Nov 03 19:15:24.594024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593517 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" Nov 03 19:15:24.594024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593609 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="48828f6f-de20-4e61-ad91-c648281f8304" containerName="istio-proxy" Nov 03 19:15:24.594024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593621 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1fe1063-da3d-4283-a161-a9ce010d56fd" containerName="istio-proxy" Nov 03 19:15:24.594024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593628 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" containerName="istio-proxy" Nov 03 19:15:24.594024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593635 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e039bad7-2408-4ef8-8d43-4e3a540a6c56" containerName="istio-proxy" Nov 03 19:15:24.594024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593640 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="85086af0-9f26-4517-88e8-8a3bd2219d2d" containerName="istio-proxy" Nov 03 19:15:24.594024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593645 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="35207ff1-165f-4250-8315-7938cbf38842" containerName="istio-proxy" Nov 03 19:15:24.594024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593651 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d614c12-4321-4857-a576-9552dc037c70" containerName="istio-proxy" Nov 03 19:15:24.594024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593658 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6f7402-2537-47a7-abc6-b21101126c4e" containerName="istio-proxy" Nov 03 19:15:24.594024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593664 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" Nov 03 19:15:24.594024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.593669 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="107848ae-227c-4b41-ab04-19b284e22fd3" containerName="istio-proxy" Nov 03 19:15:24.606627 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.604617 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" Nov 03 19:15:24.653741 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.653712 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4"] Nov 03 19:15:24.654188 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.654171 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-9dcd6c\"" Nov 03 19:15:24.654517 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.654494 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-9dcd6c-dockercfg-47jgv\"" Nov 03 19:15:24.783384 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.750917 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4\" (UID: \"d6febc76-f9af-48b5-ac02-1f7805eb1d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" Nov 03 19:15:24.783384 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.751364 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmpdc\" (UniqueName: \"kubernetes.io/projected/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-kube-api-access-gmpdc\") pod \"payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4\" (UID: \"d6febc76-f9af-48b5-ac02-1f7805eb1d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" Nov 03 19:15:24.860399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.852904 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmpdc\" (UniqueName: \"kubernetes.io/projected/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-kube-api-access-gmpdc\") pod \"payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4\" (UID: \"d6febc76-f9af-48b5-ac02-1f7805eb1d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" Nov 03 19:15:24.860399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.852984 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4\" (UID: \"d6febc76-f9af-48b5-ac02-1f7805eb1d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" Nov 03 19:15:24.860399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.854858 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4\" (UID: \"d6febc76-f9af-48b5-ac02-1f7805eb1d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" Nov 03 19:15:24.999456 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:24.999424 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x"] Nov 03 19:15:25.004457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.004435 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" Nov 03 19:15:25.005279 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.005251 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x"] Nov 03 19:15:25.007888 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.007867 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmpdc\" (UniqueName: \"kubernetes.io/projected/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-kube-api-access-gmpdc\") pod \"payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4\" (UID: \"d6febc76-f9af-48b5-ac02-1f7805eb1d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" Nov 03 19:15:25.082185 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.082154 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x\" (UID: \"7ae27eaa-ff30-4e3f-b671-5c5d429f9094\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" Nov 03 19:15:25.082372 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.082212 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9lqq\" (UniqueName: \"kubernetes.io/projected/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-kube-api-access-t9lqq\") pod \"payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x\" (UID: \"7ae27eaa-ff30-4e3f-b671-5c5d429f9094\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" Nov 03 19:15:25.184023 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.183981 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x\" (UID: \"7ae27eaa-ff30-4e3f-b671-5c5d429f9094\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" Nov 03 19:15:25.184191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.184081 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9lqq\" (UniqueName: \"kubernetes.io/projected/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-kube-api-access-t9lqq\") pod \"payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x\" (UID: \"7ae27eaa-ff30-4e3f-b671-5c5d429f9094\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" Nov 03 19:15:25.185011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.184984 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x\" (UID: \"7ae27eaa-ff30-4e3f-b671-5c5d429f9094\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" Nov 03 19:15:25.224551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.224510 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" Nov 03 19:15:25.241777 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.241746 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9lqq\" (UniqueName: \"kubernetes.io/projected/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-kube-api-access-t9lqq\") pod \"payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x\" (UID: \"7ae27eaa-ff30-4e3f-b671-5c5d429f9094\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" Nov 03 19:15:25.335592 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.335300 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" Nov 03 19:15:25.601853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.601826 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4"] Nov 03 19:15:25.602147 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:15:25.602113 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6febc76_f9af_48b5_ac02_1f7805eb1d0a.slice/crio-b1eba7116c37b5f2d268e246beaa5a18ec46dd6c676f7ed9f30da1d180bd7388 WatchSource:0}: Error finding container b1eba7116c37b5f2d268e246beaa5a18ec46dd6c676f7ed9f30da1d180bd7388: Status 404 returned error can't find the container with id b1eba7116c37b5f2d268e246beaa5a18ec46dd6c676f7ed9f30da1d180bd7388 Nov 03 19:15:25.642689 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.642661 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" event={"ID":"d6febc76-f9af-48b5-ac02-1f7805eb1d0a","Type":"ContainerStarted","Data":"b1eba7116c37b5f2d268e246beaa5a18ec46dd6c676f7ed9f30da1d180bd7388"} Nov 03 19:15:25.728257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:25.728223 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x"] Nov 03 19:15:25.729010 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:15:25.728978 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae27eaa_ff30_4e3f_b671_5c5d429f9094.slice/crio-5a14a0c2c6deb993a7ecac50f935af610c630838459e2057dffff632a7bcb850 WatchSource:0}: Error finding container 5a14a0c2c6deb993a7ecac50f935af610c630838459e2057dffff632a7bcb850: Status 404 returned error can't find the container with id 5a14a0c2c6deb993a7ecac50f935af610c630838459e2057dffff632a7bcb850 Nov 03 19:15:26.151544 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:26.151516 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d614c12-4321-4857-a576-9552dc037c70" path="/var/lib/kubelet/pods/4d614c12-4321-4857-a576-9552dc037c70/volumes" Nov 03 19:15:26.151973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:26.151959 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc320ff-47d9-44dc-b85f-27848a66a7ec" path="/var/lib/kubelet/pods/9fc320ff-47d9-44dc-b85f-27848a66a7ec/volumes" Nov 03 19:15:26.653002 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:26.652960 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" event={"ID":"d6febc76-f9af-48b5-ac02-1f7805eb1d0a","Type":"ContainerStarted","Data":"0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91"} Nov 03 19:15:26.653183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:26.653037 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" Nov 03 19:15:26.654798 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:26.654773 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" event={"ID":"7ae27eaa-ff30-4e3f-b671-5c5d429f9094","Type":"ContainerStarted","Data":"37e5e03739d3d549d16cafa36382e601accbde4d31ecd9384932d6b9e0de82b4"} Nov 03 19:15:26.654946 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:26.654818 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" event={"ID":"7ae27eaa-ff30-4e3f-b671-5c5d429f9094","Type":"ContainerStarted","Data":"5a14a0c2c6deb993a7ecac50f935af610c630838459e2057dffff632a7bcb850"} Nov 03 19:15:26.654946 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:26.654839 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" Nov 03 19:15:26.762481 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:26.762425 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" podStartSLOduration=2.7624121329999998 podStartE2EDuration="2.762412133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:26.761069373 +0000 UTC m=+1860.733630556" watchObservedRunningTime="2026-11-03 19:15:26.762412133 +0000 UTC m=+1860.734973340" Nov 03 19:15:26.874737 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:26.874682 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" podStartSLOduration=2.87466503 podStartE2EDuration="2.87466503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:26.87400331 +0000 UTC m=+1860.846564517" watchObservedRunningTime="2026-11-03 19:15:26.87466503 +0000 UTC m=+1860.847226211" Nov 03 19:15:30.160299 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.160067 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g"] Nov 03 19:15:30.166766 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.166743 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" Nov 03 19:15:30.195573 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.195549 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g"] Nov 03 19:15:30.246635 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.246600 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4f1a90\"" Nov 03 19:15:30.246944 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.246746 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4f1a90-dockercfg-4tj8w\"" Nov 03 19:15:30.362351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.347548 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5258c74f-5b40-44b8-8693-00ccaf309394-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g\" (UID: \"5258c74f-5b40-44b8-8693-00ccaf309394\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" Nov 03 19:15:30.362351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.347739 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2rlj\" (UniqueName: \"kubernetes.io/projected/5258c74f-5b40-44b8-8693-00ccaf309394-kube-api-access-c2rlj\") pod \"payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g\" (UID: \"5258c74f-5b40-44b8-8693-00ccaf309394\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" Nov 03 19:15:30.508016 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.499022 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2rlj\" (UniqueName: \"kubernetes.io/projected/5258c74f-5b40-44b8-8693-00ccaf309394-kube-api-access-c2rlj\") pod \"payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g\" (UID: \"5258c74f-5b40-44b8-8693-00ccaf309394\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" Nov 03 19:15:30.508016 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.499243 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5258c74f-5b40-44b8-8693-00ccaf309394-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g\" (UID: \"5258c74f-5b40-44b8-8693-00ccaf309394\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" Nov 03 19:15:30.508016 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.500200 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5258c74f-5b40-44b8-8693-00ccaf309394-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g\" (UID: \"5258c74f-5b40-44b8-8693-00ccaf309394\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" Nov 03 19:15:30.523632 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.523377 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp"] Nov 03 19:15:30.532802 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.532747 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp"] Nov 03 19:15:30.532969 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.532849 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" Nov 03 19:15:30.557478 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.557446 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2rlj\" (UniqueName: \"kubernetes.io/projected/5258c74f-5b40-44b8-8693-00ccaf309394-kube-api-access-c2rlj\") pod \"payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g\" (UID: \"5258c74f-5b40-44b8-8693-00ccaf309394\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" Nov 03 19:15:30.600457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.600428 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f63cce-6768-472d-bdc7-89b0ef2abd56-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp\" (UID: \"d6f63cce-6768-472d-bdc7-89b0ef2abd56\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" Nov 03 19:15:30.600592 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.600498 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprxx\" (UniqueName: \"kubernetes.io/projected/d6f63cce-6768-472d-bdc7-89b0ef2abd56-kube-api-access-sprxx\") pod \"payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp\" (UID: \"d6f63cce-6768-472d-bdc7-89b0ef2abd56\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" Nov 03 19:15:30.701854 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.701824 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f63cce-6768-472d-bdc7-89b0ef2abd56-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp\" (UID: \"d6f63cce-6768-472d-bdc7-89b0ef2abd56\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" Nov 03 19:15:30.702053 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.701907 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprxx\" (UniqueName: \"kubernetes.io/projected/d6f63cce-6768-472d-bdc7-89b0ef2abd56-kube-api-access-sprxx\") pod \"payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp\" (UID: \"d6f63cce-6768-472d-bdc7-89b0ef2abd56\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" Nov 03 19:15:30.702648 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.702624 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f63cce-6768-472d-bdc7-89b0ef2abd56-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp\" (UID: \"d6f63cce-6768-472d-bdc7-89b0ef2abd56\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" Nov 03 19:15:30.759780 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.759715 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprxx\" (UniqueName: \"kubernetes.io/projected/d6f63cce-6768-472d-bdc7-89b0ef2abd56-kube-api-access-sprxx\") pod \"payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp\" (UID: \"d6f63cce-6768-472d-bdc7-89b0ef2abd56\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" Nov 03 19:15:30.782110 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.782084 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" Nov 03 19:15:30.848703 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:30.848669 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" Nov 03 19:15:31.156672 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:31.156578 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g"] Nov 03 19:15:31.157000 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:15:31.156969 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5258c74f_5b40_44b8_8693_00ccaf309394.slice/crio-edbc928ccd47003f635853e64c768a6fb807587925caf19e8db79ea819e491f4 WatchSource:0}: Error finding container edbc928ccd47003f635853e64c768a6fb807587925caf19e8db79ea819e491f4: Status 404 returned error can't find the container with id edbc928ccd47003f635853e64c768a6fb807587925caf19e8db79ea819e491f4 Nov 03 19:15:31.261493 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:31.261467 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp"] Nov 03 19:15:31.261989 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:15:31.261957 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f63cce_6768_472d_bdc7_89b0ef2abd56.slice/crio-f62925eec488ac82be693bf086aeeba184c143ee9fd2568d4a07c07d5a0206a0 WatchSource:0}: Error finding container f62925eec488ac82be693bf086aeeba184c143ee9fd2568d4a07c07d5a0206a0: Status 404 returned error can't find the container with id f62925eec488ac82be693bf086aeeba184c143ee9fd2568d4a07c07d5a0206a0 Nov 03 19:15:31.702985 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:31.702948 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" event={"ID":"5258c74f-5b40-44b8-8693-00ccaf309394","Type":"ContainerStarted","Data":"27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820"} Nov 03 19:15:31.703183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:31.703009 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" event={"ID":"5258c74f-5b40-44b8-8693-00ccaf309394","Type":"ContainerStarted","Data":"edbc928ccd47003f635853e64c768a6fb807587925caf19e8db79ea819e491f4"} Nov 03 19:15:31.703183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:31.703057 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" Nov 03 19:15:31.705180 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:31.705152 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" event={"ID":"d6f63cce-6768-472d-bdc7-89b0ef2abd56","Type":"ContainerStarted","Data":"76b7aa35b56c7c8e043fb11e0da5a73767d53cdfd29e844675dd489a995214f3"} Nov 03 19:15:31.705318 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:31.705202 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" event={"ID":"d6f63cce-6768-472d-bdc7-89b0ef2abd56","Type":"ContainerStarted","Data":"f62925eec488ac82be693bf086aeeba184c143ee9fd2568d4a07c07d5a0206a0"} Nov 03 19:15:31.705318 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:31.705244 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" Nov 03 19:15:31.813444 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:31.813388 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" podStartSLOduration=1.813374249 podStartE2EDuration="1.813374249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:31.81051493 +0000 UTC m=+1865.783076112" watchObservedRunningTime="2026-11-03 19:15:31.813374249 +0000 UTC m=+1865.785935430" Nov 03 19:15:31.920420 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:31.920366 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" podStartSLOduration=1.920350816 podStartE2EDuration="1.920350816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:31.918165739 +0000 UTC m=+1865.890726921" watchObservedRunningTime="2026-11-03 19:15:31.920350816 +0000 UTC m=+1865.892911995" Nov 03 19:15:37.666578 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:37.666543 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" Nov 03 19:15:37.667134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:37.666950 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" Nov 03 19:15:41.392057 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:15:41.392010 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:15:42.715931 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:42.715888 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" Nov 03 19:15:42.716410 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:42.715964 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" Nov 03 19:15:46.813350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.812830 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw"] Nov 03 19:15:46.843476 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.842140 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:46.950357 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.941760 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dd1327-openshift-default-dockercfg-l4b6m\"" Nov 03 19:15:46.976404 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.970713 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw"] Nov 03 19:15:46.996502 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.996465 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-envoy\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:46.996818 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.996799 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-podinfo\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:46.996973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.996957 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-token\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:46.997122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.997108 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2gvs\" (UniqueName: \"kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-kube-api-access-l2gvs\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:46.997237 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.997225 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-socket\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:46.997356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.997342 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-credential-socket\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:46.997510 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.997492 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istiod-ca-cert\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:46.997660 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.997645 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-data\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:46.998673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:46.997799 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-certs\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.154442 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.101282 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2gvs\" (UniqueName: \"kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-kube-api-access-l2gvs\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.154785 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.154749 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-socket\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.154992 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.154972 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-credential-socket\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.155220 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.155200 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istiod-ca-cert\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.155448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.155429 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-data\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.155645 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.155629 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-certs\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.155914 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.155898 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-envoy\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.156048 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.156031 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-podinfo\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.165353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.156204 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-token\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.167476 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.156280 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-socket\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.167713 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.156031 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-credential-socket\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.168043 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.167989 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-data\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.169028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.169006 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istiod-ca-cert\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.169289 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.169268 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-certs\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.186347 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.169822 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-envoy\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.218929 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.218878 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-podinfo\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.219633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.219589 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2gvs\" (UniqueName: \"kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-kube-api-access-l2gvs\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.253300 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.245245 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-token\") pod \"e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.392394 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.392076 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:47.820227 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.820194 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw"] Nov 03 19:15:47.820611 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:15:47.820540 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99b4137_3f69_4ca3_b3c2_e3c3a33f6996.slice/crio-af15a50ef867bd20f2d41c44a8e35616fdf5e73feed457cf61cc125d00c9119e WatchSource:0}: Error finding container af15a50ef867bd20f2d41c44a8e35616fdf5e73feed457cf61cc125d00c9119e: Status 404 returned error can't find the container with id af15a50ef867bd20f2d41c44a8e35616fdf5e73feed457cf61cc125d00c9119e Nov 03 19:15:47.823564 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.823531 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:47.823640 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.823598 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:47.823640 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:47.823633 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:48.010404 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:48.008150 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" event={"ID":"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996","Type":"ContainerStarted","Data":"77319ec8bfb0a30b0c793c586cae4beea50580b908a35be3482660502ada0ec1"} Nov 03 19:15:48.010404 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:48.008208 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" event={"ID":"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996","Type":"ContainerStarted","Data":"af15a50ef867bd20f2d41c44a8e35616fdf5e73feed457cf61cc125d00c9119e"} Nov 03 19:15:48.129161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:48.127647 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" podStartSLOduration=2.127626497 podStartE2EDuration="2.127626497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:48.123444118 +0000 UTC m=+1882.096005301" watchObservedRunningTime="2026-11-03 19:15:48.127626497 +0000 UTC m=+1882.100187679" Nov 03 19:15:48.397405 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:48.396446 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:48.407349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:48.398935 5019 patch_prober.go:29] interesting pod/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:15:48.407349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:48.399087 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 19:15:49.423618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:49.423590 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:50.023966 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:50.023925 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:50.025092 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:50.025068 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:15:50.975373 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:50.975312 5019 patch_prober.go:29] interesting pod/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:15:50.975807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:50.975408 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:15:53.048074 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:53.048043 5019 generic.go:356] "Generic (PLEG): container finished" podID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerID="3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e" exitCode=0 Nov 03 19:15:53.048616 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:53.048109 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" event={"ID":"dd9a1357-9f0e-4616-8e9a-6c60f4385cec","Type":"ContainerDied","Data":"3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e"} Nov 03 19:15:53.048616 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:53.048542 5019 scope.go:122] "RemoveContainer" containerID="3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e" Nov 03 19:15:53.050221 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:53.050188 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:53.050293 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:53.050265 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:53.050356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:53.050295 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:15:54.102934 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:54.102885 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" event={"ID":"dd9a1357-9f0e-4616-8e9a-6c60f4385cec","Type":"ContainerStarted","Data":"46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d"} Nov 03 19:15:54.339654 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:54.339616 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:15:54.339976 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:54.339948 5019 patch_prober.go:29] interesting pod/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:15:54.340031 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:54.340011 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:15:54.812349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:54.811346 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp"] Nov 03 19:15:54.827348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:54.824543 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" Nov 03 19:15:54.877796 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:54.877759 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-dd1327\"" Nov 03 19:15:54.880194 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:54.880168 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-dd1327-dockercfg-thgg8\"" Nov 03 19:15:54.887217 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:54.887187 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9b253d-9da9-4b78-8894-860f9645f1b1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp\" (UID: \"ee9b253d-9da9-4b78-8894-860f9645f1b1\") " pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" Nov 03 19:15:54.887436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:54.887417 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbmmx\" (UniqueName: \"kubernetes.io/projected/ee9b253d-9da9-4b78-8894-860f9645f1b1-kube-api-access-jbmmx\") pod \"payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp\" (UID: \"ee9b253d-9da9-4b78-8894-860f9645f1b1\") " pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" Nov 03 19:15:54.922363 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:54.921618 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp"] Nov 03 19:15:55.031560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.031522 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc"] Nov 03 19:15:55.037957 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.036444 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9b253d-9da9-4b78-8894-860f9645f1b1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp\" (UID: \"ee9b253d-9da9-4b78-8894-860f9645f1b1\") " pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" Nov 03 19:15:55.037957 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.036488 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbmmx\" (UniqueName: \"kubernetes.io/projected/ee9b253d-9da9-4b78-8894-860f9645f1b1-kube-api-access-jbmmx\") pod \"payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp\" (UID: \"ee9b253d-9da9-4b78-8894-860f9645f1b1\") " pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" Nov 03 19:15:55.037957 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.037712 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9b253d-9da9-4b78-8894-860f9645f1b1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp\" (UID: \"ee9b253d-9da9-4b78-8894-860f9645f1b1\") " pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" Nov 03 19:15:55.045459 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.045433 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" Nov 03 19:15:55.048725 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.048686 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc"] Nov 03 19:15:55.107776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.107274 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbmmx\" (UniqueName: \"kubernetes.io/projected/ee9b253d-9da9-4b78-8894-860f9645f1b1-kube-api-access-jbmmx\") pod \"payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp\" (UID: \"ee9b253d-9da9-4b78-8894-860f9645f1b1\") " pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" Nov 03 19:15:55.192481 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.192445 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" Nov 03 19:15:55.240243 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.239687 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ec3cf1-6c12-41b3-9c86-7849471eb005-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc\" (UID: \"97ec3cf1-6c12-41b3-9c86-7849471eb005\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" Nov 03 19:15:55.240243 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.239837 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twv78\" (UniqueName: \"kubernetes.io/projected/97ec3cf1-6c12-41b3-9c86-7849471eb005-kube-api-access-twv78\") pod \"payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc\" (UID: \"97ec3cf1-6c12-41b3-9c86-7849471eb005\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" Nov 03 19:15:55.342451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.342415 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ec3cf1-6c12-41b3-9c86-7849471eb005-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc\" (UID: \"97ec3cf1-6c12-41b3-9c86-7849471eb005\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" Nov 03 19:15:55.342636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.342562 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twv78\" (UniqueName: \"kubernetes.io/projected/97ec3cf1-6c12-41b3-9c86-7849471eb005-kube-api-access-twv78\") pod \"payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc\" (UID: \"97ec3cf1-6c12-41b3-9c86-7849471eb005\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" Nov 03 19:15:55.344473 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.344149 5019 patch_prober.go:29] interesting pod/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:15:55.344473 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.344198 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:15:55.344473 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.344464 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ec3cf1-6c12-41b3-9c86-7849471eb005-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc\" (UID: \"97ec3cf1-6c12-41b3-9c86-7849471eb005\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" Nov 03 19:15:55.403386 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.403301 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twv78\" (UniqueName: \"kubernetes.io/projected/97ec3cf1-6c12-41b3-9c86-7849471eb005-kube-api-access-twv78\") pod \"payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc\" (UID: \"97ec3cf1-6c12-41b3-9c86-7849471eb005\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" Nov 03 19:15:55.415402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.414978 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" Nov 03 19:15:55.604647 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.604608 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp"] Nov 03 19:15:55.626162 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:15:55.626132 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9b253d_9da9_4b78_8894_860f9645f1b1.slice/crio-275a4655731105f8ace84a755777d538e500a1702d688839f789f82d8d8f1192 WatchSource:0}: Error finding container 275a4655731105f8ace84a755777d538e500a1702d688839f789f82d8d8f1192: Status 404 returned error can't find the container with id 275a4655731105f8ace84a755777d538e500a1702d688839f789f82d8d8f1192 Nov 03 19:15:55.835760 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:15:55.835737 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ec3cf1_6c12_41b3_9c86_7849471eb005.slice/crio-65f1dfd041b6c6ad6c9e2a101dbaa0855bef5b21f0d7e54f242f0b5711567470 WatchSource:0}: Error finding container 65f1dfd041b6c6ad6c9e2a101dbaa0855bef5b21f0d7e54f242f0b5711567470: Status 404 returned error can't find the container with id 65f1dfd041b6c6ad6c9e2a101dbaa0855bef5b21f0d7e54f242f0b5711567470 Nov 03 19:15:55.836689 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:55.836502 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc"] Nov 03 19:15:56.124455 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.124421 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" event={"ID":"97ec3cf1-6c12-41b3-9c86-7849471eb005","Type":"ContainerStarted","Data":"bb07809319d7cc160ddd28547f29ab95cf2cac20648e2326ac036d6a43ca6c8b"} Nov 03 19:15:56.124901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.124480 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" event={"ID":"97ec3cf1-6c12-41b3-9c86-7849471eb005","Type":"ContainerStarted","Data":"65f1dfd041b6c6ad6c9e2a101dbaa0855bef5b21f0d7e54f242f0b5711567470"} Nov 03 19:15:56.124901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.124642 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" Nov 03 19:15:56.126618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.126592 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" event={"ID":"ee9b253d-9da9-4b78-8894-860f9645f1b1","Type":"ContainerStarted","Data":"5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983"} Nov 03 19:15:56.126728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.126641 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" event={"ID":"ee9b253d-9da9-4b78-8894-860f9645f1b1","Type":"ContainerStarted","Data":"275a4655731105f8ace84a755777d538e500a1702d688839f789f82d8d8f1192"} Nov 03 19:15:56.126728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.126702 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" Nov 03 19:15:56.250228 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.250117 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" podStartSLOduration=2.25009996 podStartE2EDuration="2.25009996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:56.248994773 +0000 UTC m=+1890.221555954" watchObservedRunningTime="2026-11-03 19:15:56.25009996 +0000 UTC m=+1890.222661153" Nov 03 19:15:56.339323 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.339294 5019 patch_prober.go:29] interesting pod/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:15:56.339516 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.339355 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:15:56.648214 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.641275 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" podStartSLOduration=2.641254976 podStartE2EDuration="2.641254976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:56.37259167 +0000 UTC m=+1890.345152855" watchObservedRunningTime="2026-11-03 19:15:56.641254976 +0000 UTC m=+1890.613816159" Nov 03 19:15:56.648214 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.644204 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8875-920945"] Nov 03 19:15:56.649871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.649848 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-920945" Nov 03 19:15:56.708631 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.704867 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8875-920945"] Nov 03 19:15:56.708631 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.708484 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2f8z4\"" Nov 03 19:15:56.718318 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.718276 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndxcp\" (UniqueName: \"kubernetes.io/projected/742a6cb1-4d3a-4994-b0b3-bd9501117ca0-kube-api-access-ndxcp\") pod \"test-curl-8875-920945\" (UID: \"742a6cb1-4d3a-4994-b0b3-bd9501117ca0\") " pod="opendatahub/test-curl-8875-920945" Nov 03 19:15:56.819939 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.819902 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndxcp\" (UniqueName: \"kubernetes.io/projected/742a6cb1-4d3a-4994-b0b3-bd9501117ca0-kube-api-access-ndxcp\") pod \"test-curl-8875-920945\" (UID: \"742a6cb1-4d3a-4994-b0b3-bd9501117ca0\") " pod="opendatahub/test-curl-8875-920945" Nov 03 19:15:56.897922 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:56.897889 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndxcp\" (UniqueName: \"kubernetes.io/projected/742a6cb1-4d3a-4994-b0b3-bd9501117ca0-kube-api-access-ndxcp\") pod \"test-curl-8875-920945\" (UID: \"742a6cb1-4d3a-4994-b0b3-bd9501117ca0\") " pod="opendatahub/test-curl-8875-920945" Nov 03 19:15:57.018912 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:57.018880 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-920945" Nov 03 19:15:57.339886 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:57.339812 5019 patch_prober.go:29] interesting pod/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:15:57.339886 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:57.339865 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:15:57.424951 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:57.424916 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8875-920945"] Nov 03 19:15:57.426027 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:15:57.425996 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod742a6cb1_4d3a_4994_b0b3_bd9501117ca0.slice/crio-6da711a2515fd12b4e9b755fd9b955d16acfaedb94c0976017dc2fffc4812b2a WatchSource:0}: Error finding container 6da711a2515fd12b4e9b755fd9b955d16acfaedb94c0976017dc2fffc4812b2a: Status 404 returned error can't find the container with id 6da711a2515fd12b4e9b755fd9b955d16acfaedb94c0976017dc2fffc4812b2a Nov 03 19:15:57.428136 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:57.428117 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:15:58.150613 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:58.150583 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8875-920945" event={"ID":"742a6cb1-4d3a-4994-b0b3-bd9501117ca0","Type":"ContainerStarted","Data":"6da711a2515fd12b4e9b755fd9b955d16acfaedb94c0976017dc2fffc4812b2a"} Nov 03 19:15:58.339185 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:58.339144 5019 patch_prober.go:29] interesting pod/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:15:58.339420 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:58.339203 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:15:59.339370 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:59.339284 5019 patch_prober.go:29] interesting pod/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:15:59.339933 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:15:59.339386 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:16:00.344055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:00.344021 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:16:00.345403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:00.345381 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:16:00.346154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:00.346133 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:16:01.170752 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:01.170715 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8875-920945" event={"ID":"742a6cb1-4d3a-4994-b0b3-bd9501117ca0","Type":"ContainerStarted","Data":"0235522b49357a1f97213338c031e2cae7945dd39393d9934e0cdd112ea0bfa5"} Nov 03 19:16:01.277553 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:01.277497 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8875-920945" podStartSLOduration=2.01940675 podStartE2EDuration="5.277484022s" totalImagesPullingTime="3.258077272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:01.276823321 +0000 UTC m=+1895.249384503" watchObservedRunningTime="2026-11-03 19:16:01.277484022 +0000 UTC m=+1895.250045203" Nov 03 19:16:02.736270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:02.736238 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8875-920945"] Nov 03 19:16:02.741122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:02.741090 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8875-920945"] Nov 03 19:16:02.887748 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:02.887713 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8875-8e8798"] Nov 03 19:16:02.888473 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:02.888457 5019 cpu_manager.go:414] "containerMap: removing container" podUID="742a6cb1-4d3a-4994-b0b3-bd9501117ca0" containerName="test-curl-8875-920945" Nov 03 19:16:02.888527 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:02.888475 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742a6cb1-4d3a-4994-b0b3-bd9501117ca0" containerName="test-curl-8875-920945" Nov 03 19:16:02.888633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:02.888624 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="742a6cb1-4d3a-4994-b0b3-bd9501117ca0" containerName="test-curl-8875-920945" Nov 03 19:16:02.892513 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:02.892494 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-8e8798" Nov 03 19:16:02.945189 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:02.945155 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8875-8e8798"] Nov 03 19:16:03.053065 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:03.052979 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npgdw\" (UniqueName: \"kubernetes.io/projected/2de4dc8f-b79c-4570-bade-2dd508ba8809-kube-api-access-npgdw\") pod \"test-curl-8875-8e8798\" (UID: \"2de4dc8f-b79c-4570-bade-2dd508ba8809\") " pod="opendatahub/test-curl-8875-8e8798" Nov 03 19:16:03.154593 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:03.154555 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npgdw\" (UniqueName: \"kubernetes.io/projected/2de4dc8f-b79c-4570-bade-2dd508ba8809-kube-api-access-npgdw\") pod \"test-curl-8875-8e8798\" (UID: \"2de4dc8f-b79c-4570-bade-2dd508ba8809\") " pod="opendatahub/test-curl-8875-8e8798" Nov 03 19:16:03.185037 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:03.184991 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8875-920945" podUID="742a6cb1-4d3a-4994-b0b3-bd9501117ca0" containerName="test-curl-8875-920945" containerID="cri-o://0235522b49357a1f97213338c031e2cae7945dd39393d9934e0cdd112ea0bfa5" gracePeriod=2 Nov 03 19:16:03.211821 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:03.211788 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npgdw\" (UniqueName: \"kubernetes.io/projected/2de4dc8f-b79c-4570-bade-2dd508ba8809-kube-api-access-npgdw\") pod \"test-curl-8875-8e8798\" (UID: \"2de4dc8f-b79c-4570-bade-2dd508ba8809\") " pod="opendatahub/test-curl-8875-8e8798" Nov 03 19:16:03.232850 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:03.232812 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8875-920945\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="742a6cb1-4d3a-4994-b0b3-bd9501117ca0" pod="opendatahub/test-curl-8875-920945" Nov 03 19:16:03.505921 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:03.505888 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-8e8798" Nov 03 19:16:03.882735 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:16:03.882051 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2de4dc8f_b79c_4570_bade_2dd508ba8809.slice/crio-24ee82775598d4c5820dbe929e5d13cd074ba22f06a06d26f994befd23e5e805 WatchSource:0}: Error finding container 24ee82775598d4c5820dbe929e5d13cd074ba22f06a06d26f994befd23e5e805: Status 404 returned error can't find the container with id 24ee82775598d4c5820dbe929e5d13cd074ba22f06a06d26f994befd23e5e805 Nov 03 19:16:03.885140 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:03.885110 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8875-8e8798"] Nov 03 19:16:04.193259 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:04.193219 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8875-8e8798" event={"ID":"2de4dc8f-b79c-4570-bade-2dd508ba8809","Type":"ContainerStarted","Data":"6ab57d5bcf9403ee68970c670c862522ac09ff69f91065f6c1943c1382937060"} Nov 03 19:16:04.193450 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:04.193269 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8875-8e8798" event={"ID":"2de4dc8f-b79c-4570-bade-2dd508ba8809","Type":"ContainerStarted","Data":"24ee82775598d4c5820dbe929e5d13cd074ba22f06a06d26f994befd23e5e805"} Nov 03 19:16:04.301401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:04.301314 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8875-8e8798" podStartSLOduration=2.301301305 podStartE2EDuration="2.301301305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:04.298842175 +0000 UTC m=+1898.271403357" watchObservedRunningTime="2026-11-03 19:16:04.301301305 +0000 UTC m=+1898.273862487" Nov 03 19:16:05.436343 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:05.436306 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-920945" Nov 03 19:16:05.499086 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:05.499010 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/742a6cb1-4d3a-4994-b0b3-bd9501117ca0-kube-api-access-ndxcp\" (UniqueName: \"kubernetes.io/projected/742a6cb1-4d3a-4994-b0b3-bd9501117ca0-kube-api-access-ndxcp\") pod \"742a6cb1-4d3a-4994-b0b3-bd9501117ca0\" (UID: \"742a6cb1-4d3a-4994-b0b3-bd9501117ca0\") " Nov 03 19:16:05.501193 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:05.501163 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742a6cb1-4d3a-4994-b0b3-bd9501117ca0-kube-api-access-ndxcp" pod "742a6cb1-4d3a-4994-b0b3-bd9501117ca0" (UID: "742a6cb1-4d3a-4994-b0b3-bd9501117ca0"). InnerVolumeSpecName "kube-api-access-ndxcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:05.601198 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:05.601170 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndxcp\" (UniqueName: \"kubernetes.io/projected/742a6cb1-4d3a-4994-b0b3-bd9501117ca0-kube-api-access-ndxcp\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:05.852409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:05.852311 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8875-8e8798"] Nov 03 19:16:05.857141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:05.857107 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8875-8e8798"] Nov 03 19:16:06.002745 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.002709 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8875-996e09"] Nov 03 19:16:06.003626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.003592 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2de4dc8f-b79c-4570-bade-2dd508ba8809" containerName="test-curl-8875-8e8798" Nov 03 19:16:06.003626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.003616 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de4dc8f-b79c-4570-bade-2dd508ba8809" containerName="test-curl-8875-8e8798" Nov 03 19:16:06.003873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.003858 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de4dc8f-b79c-4570-bade-2dd508ba8809" containerName="test-curl-8875-8e8798" Nov 03 19:16:06.007772 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.007751 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-996e09" Nov 03 19:16:06.059985 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.059961 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8875-996e09"] Nov 03 19:16:06.108882 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.108773 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k9rh\" (UniqueName: \"kubernetes.io/projected/ab7c7403-6027-4cec-a6b7-82cca4bb8781-kube-api-access-9k9rh\") pod \"test-curl-8875-996e09\" (UID: \"ab7c7403-6027-4cec-a6b7-82cca4bb8781\") " pod="opendatahub/test-curl-8875-996e09" Nov 03 19:16:06.152831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.152800 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="742a6cb1-4d3a-4994-b0b3-bd9501117ca0" path="/var/lib/kubelet/pods/742a6cb1-4d3a-4994-b0b3-bd9501117ca0/volumes" Nov 03 19:16:06.209231 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.209200 5019 generic.go:356] "Generic (PLEG): container finished" podID="742a6cb1-4d3a-4994-b0b3-bd9501117ca0" containerID="0235522b49357a1f97213338c031e2cae7945dd39393d9934e0cdd112ea0bfa5" exitCode=137 Nov 03 19:16:06.209441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.209259 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-920945" Nov 03 19:16:06.209441 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.209306 5019 scope.go:122] "RemoveContainer" containerID="0235522b49357a1f97213338c031e2cae7945dd39393d9934e0cdd112ea0bfa5" Nov 03 19:16:06.209607 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.209542 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8875-8e8798" podUID="2de4dc8f-b79c-4570-bade-2dd508ba8809" containerName="test-curl-8875-8e8798" containerID="cri-o://6ab57d5bcf9403ee68970c670c862522ac09ff69f91065f6c1943c1382937060" gracePeriod=2 Nov 03 19:16:06.210227 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.210106 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9rh\" (UniqueName: \"kubernetes.io/projected/ab7c7403-6027-4cec-a6b7-82cca4bb8781-kube-api-access-9k9rh\") pod \"test-curl-8875-996e09\" (UID: \"ab7c7403-6027-4cec-a6b7-82cca4bb8781\") " pod="opendatahub/test-curl-8875-996e09" Nov 03 19:16:06.220558 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.220541 5019 scope.go:122] "RemoveContainer" containerID="0235522b49357a1f97213338c031e2cae7945dd39393d9934e0cdd112ea0bfa5" Nov 03 19:16:06.220819 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:06.220798 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0235522b49357a1f97213338c031e2cae7945dd39393d9934e0cdd112ea0bfa5\": container with ID starting with 0235522b49357a1f97213338c031e2cae7945dd39393d9934e0cdd112ea0bfa5 not found: ID does not exist" containerID="0235522b49357a1f97213338c031e2cae7945dd39393d9934e0cdd112ea0bfa5" Nov 03 19:16:06.220888 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.220826 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0235522b49357a1f97213338c031e2cae7945dd39393d9934e0cdd112ea0bfa5"} err="failed to get container status \"0235522b49357a1f97213338c031e2cae7945dd39393d9934e0cdd112ea0bfa5\": rpc error: code = NotFound desc = could not find container \"0235522b49357a1f97213338c031e2cae7945dd39393d9934e0cdd112ea0bfa5\": container with ID starting with 0235522b49357a1f97213338c031e2cae7945dd39393d9934e0cdd112ea0bfa5 not found: ID does not exist" Nov 03 19:16:06.268288 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.268257 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9rh\" (UniqueName: \"kubernetes.io/projected/ab7c7403-6027-4cec-a6b7-82cca4bb8781-kube-api-access-9k9rh\") pod \"test-curl-8875-996e09\" (UID: \"ab7c7403-6027-4cec-a6b7-82cca4bb8781\") " pod="opendatahub/test-curl-8875-996e09" Nov 03 19:16:06.319956 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.319932 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-996e09" Nov 03 19:16:06.693702 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:06.693670 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8875-996e09"] Nov 03 19:16:06.694272 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:16:06.694245 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab7c7403_6027_4cec_a6b7_82cca4bb8781.slice/crio-036033e190806e9589cdfce3a064c296cc7e9bd53405ddf6a7910ea474502e64 WatchSource:0}: Error finding container 036033e190806e9589cdfce3a064c296cc7e9bd53405ddf6a7910ea474502e64: Status 404 returned error can't find the container with id 036033e190806e9589cdfce3a064c296cc7e9bd53405ddf6a7910ea474502e64 Nov 03 19:16:07.137427 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:07.137343 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" Nov 03 19:16:07.137577 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:07.137563 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" Nov 03 19:16:07.220596 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:07.220561 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8875-996e09" event={"ID":"ab7c7403-6027-4cec-a6b7-82cca4bb8781","Type":"ContainerStarted","Data":"9dbe60b687417ffad7206457dbf6b4d0a5f0c91bf93eace35993244a0ab91d8f"} Nov 03 19:16:07.220910 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:07.220601 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8875-996e09" event={"ID":"ab7c7403-6027-4cec-a6b7-82cca4bb8781","Type":"ContainerStarted","Data":"036033e190806e9589cdfce3a064c296cc7e9bd53405ddf6a7910ea474502e64"} Nov 03 19:16:07.463170 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:07.463105 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8875-996e09" podStartSLOduration=2.463086945 podStartE2EDuration="2.463086945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:07.460052574 +0000 UTC m=+1901.432613756" watchObservedRunningTime="2026-11-03 19:16:07.463086945 +0000 UTC m=+1901.435648128" Nov 03 19:16:08.444172 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:08.444152 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-8e8798" Nov 03 19:16:08.543039 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:08.542954 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de4dc8f-b79c-4570-bade-2dd508ba8809-kube-api-access-npgdw\" (UniqueName: \"kubernetes.io/projected/2de4dc8f-b79c-4570-bade-2dd508ba8809-kube-api-access-npgdw\") pod \"2de4dc8f-b79c-4570-bade-2dd508ba8809\" (UID: \"2de4dc8f-b79c-4570-bade-2dd508ba8809\") " Nov 03 19:16:08.545272 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:08.545241 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de4dc8f-b79c-4570-bade-2dd508ba8809-kube-api-access-npgdw" pod "2de4dc8f-b79c-4570-bade-2dd508ba8809" (UID: "2de4dc8f-b79c-4570-bade-2dd508ba8809"). InnerVolumeSpecName "kube-api-access-npgdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:08.644502 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:08.644466 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npgdw\" (UniqueName: \"kubernetes.io/projected/2de4dc8f-b79c-4570-bade-2dd508ba8809-kube-api-access-npgdw\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:08.879663 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:08.879586 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8875-996e09"] Nov 03 19:16:08.882882 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:08.882859 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8875-996e09"] Nov 03 19:16:09.034238 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.034209 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8875-a22504"] Nov 03 19:16:09.035022 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.035004 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ab7c7403-6027-4cec-a6b7-82cca4bb8781" containerName="test-curl-8875-996e09" Nov 03 19:16:09.035078 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.035024 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7c7403-6027-4cec-a6b7-82cca4bb8781" containerName="test-curl-8875-996e09" Nov 03 19:16:09.035137 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.035130 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7c7403-6027-4cec-a6b7-82cca4bb8781" containerName="test-curl-8875-996e09" Nov 03 19:16:09.039034 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.039015 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-a22504" Nov 03 19:16:09.092868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.092840 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8875-a22504"] Nov 03 19:16:09.153101 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.153020 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qhd7\" (UniqueName: \"kubernetes.io/projected/f9dc02d7-fdf4-4454-a315-977cb9abbecf-kube-api-access-6qhd7\") pod \"test-curl-8875-a22504\" (UID: \"f9dc02d7-fdf4-4454-a315-977cb9abbecf\") " pod="opendatahub/test-curl-8875-a22504" Nov 03 19:16:09.237225 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.237190 5019 generic.go:356] "Generic (PLEG): container finished" podID="2de4dc8f-b79c-4570-bade-2dd508ba8809" containerID="6ab57d5bcf9403ee68970c670c862522ac09ff69f91065f6c1943c1382937060" exitCode=137 Nov 03 19:16:09.237448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.237258 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-8e8798" Nov 03 19:16:09.237448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.237276 5019 scope.go:122] "RemoveContainer" containerID="6ab57d5bcf9403ee68970c670c862522ac09ff69f91065f6c1943c1382937060" Nov 03 19:16:09.237585 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.237544 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8875-996e09" podUID="ab7c7403-6027-4cec-a6b7-82cca4bb8781" containerName="test-curl-8875-996e09" containerID="cri-o://9dbe60b687417ffad7206457dbf6b4d0a5f0c91bf93eace35993244a0ab91d8f" gracePeriod=2 Nov 03 19:16:09.248278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.248262 5019 scope.go:122] "RemoveContainer" containerID="6ab57d5bcf9403ee68970c670c862522ac09ff69f91065f6c1943c1382937060" Nov 03 19:16:09.248592 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:09.248571 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab57d5bcf9403ee68970c670c862522ac09ff69f91065f6c1943c1382937060\": container with ID starting with 6ab57d5bcf9403ee68970c670c862522ac09ff69f91065f6c1943c1382937060 not found: ID does not exist" containerID="6ab57d5bcf9403ee68970c670c862522ac09ff69f91065f6c1943c1382937060" Nov 03 19:16:09.248660 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.248599 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab57d5bcf9403ee68970c670c862522ac09ff69f91065f6c1943c1382937060"} err="failed to get container status \"6ab57d5bcf9403ee68970c670c862522ac09ff69f91065f6c1943c1382937060\": rpc error: code = NotFound desc = could not find container \"6ab57d5bcf9403ee68970c670c862522ac09ff69f91065f6c1943c1382937060\": container with ID starting with 6ab57d5bcf9403ee68970c670c862522ac09ff69f91065f6c1943c1382937060 not found: ID does not exist" Nov 03 19:16:09.254054 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.254034 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qhd7\" (UniqueName: \"kubernetes.io/projected/f9dc02d7-fdf4-4454-a315-977cb9abbecf-kube-api-access-6qhd7\") pod \"test-curl-8875-a22504\" (UID: \"f9dc02d7-fdf4-4454-a315-977cb9abbecf\") " pod="opendatahub/test-curl-8875-a22504" Nov 03 19:16:09.285484 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:09.285430 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8875-996e09\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="ab7c7403-6027-4cec-a6b7-82cca4bb8781" pod="opendatahub/test-curl-8875-996e09" Nov 03 19:16:09.311182 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.311153 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qhd7\" (UniqueName: \"kubernetes.io/projected/f9dc02d7-fdf4-4454-a315-977cb9abbecf-kube-api-access-6qhd7\") pod \"test-curl-8875-a22504\" (UID: \"f9dc02d7-fdf4-4454-a315-977cb9abbecf\") " pod="opendatahub/test-curl-8875-a22504" Nov 03 19:16:09.333067 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:09.333034 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8875-996e09\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="ab7c7403-6027-4cec-a6b7-82cca4bb8781" pod="opendatahub/test-curl-8875-996e09" Nov 03 19:16:09.351886 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.351863 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-a22504" Nov 03 19:16:09.380678 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:09.380637 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8875-8e8798\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="2de4dc8f-b79c-4570-bade-2dd508ba8809" pod="opendatahub/test-curl-8875-8e8798" Nov 03 19:16:09.755391 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:09.755359 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8875-a22504"] Nov 03 19:16:09.756269 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:16:09.756240 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9dc02d7_fdf4_4454_a315_977cb9abbecf.slice/crio-db2e729adb3217327dc71cbc066e4e93806b0d903f95a2c79fc086b0f7cf57e3 WatchSource:0}: Error finding container db2e729adb3217327dc71cbc066e4e93806b0d903f95a2c79fc086b0f7cf57e3: Status 404 returned error can't find the container with id db2e729adb3217327dc71cbc066e4e93806b0d903f95a2c79fc086b0f7cf57e3 Nov 03 19:16:10.153377 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:10.153299 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de4dc8f-b79c-4570-bade-2dd508ba8809" path="/var/lib/kubelet/pods/2de4dc8f-b79c-4570-bade-2dd508ba8809/volumes" Nov 03 19:16:10.246952 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:10.246921 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8875-a22504" event={"ID":"f9dc02d7-fdf4-4454-a315-977cb9abbecf","Type":"ContainerStarted","Data":"89d87ac8d9c2c9d7059b61deb9f47e33bf2e97b6542fb64d1b5eaf4dc97d9191"} Nov 03 19:16:10.247158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:10.246963 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8875-a22504" event={"ID":"f9dc02d7-fdf4-4454-a315-977cb9abbecf","Type":"ContainerStarted","Data":"db2e729adb3217327dc71cbc066e4e93806b0d903f95a2c79fc086b0f7cf57e3"} Nov 03 19:16:10.355552 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:10.355501 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8875-a22504" podStartSLOduration=2.355486499 podStartE2EDuration="2.355486499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:10.352310706 +0000 UTC m=+1904.324871885" watchObservedRunningTime="2026-11-03 19:16:10.355486499 +0000 UTC m=+1904.328047680" Nov 03 19:16:11.305962 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:11.305929 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:16:11.508201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:11.508141 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-996e09" Nov 03 19:16:11.590424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:11.590396 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7c7403-6027-4cec-a6b7-82cca4bb8781-kube-api-access-9k9rh\" (UniqueName: \"kubernetes.io/projected/ab7c7403-6027-4cec-a6b7-82cca4bb8781-kube-api-access-9k9rh\") pod \"ab7c7403-6027-4cec-a6b7-82cca4bb8781\" (UID: \"ab7c7403-6027-4cec-a6b7-82cca4bb8781\") " Nov 03 19:16:11.592933 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:11.592896 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7c7403-6027-4cec-a6b7-82cca4bb8781-kube-api-access-9k9rh" pod "ab7c7403-6027-4cec-a6b7-82cca4bb8781" (UID: "ab7c7403-6027-4cec-a6b7-82cca4bb8781"). InnerVolumeSpecName "kube-api-access-9k9rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:11.692602 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:11.692576 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k9rh\" (UniqueName: \"kubernetes.io/projected/ab7c7403-6027-4cec-a6b7-82cca4bb8781-kube-api-access-9k9rh\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:11.839637 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:11.839554 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8875-a22504"] Nov 03 19:16:11.843848 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:11.843817 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8875-a22504"] Nov 03 19:16:12.151520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:12.151448 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7c7403-6027-4cec-a6b7-82cca4bb8781" path="/var/lib/kubelet/pods/ab7c7403-6027-4cec-a6b7-82cca4bb8781/volumes" Nov 03 19:16:12.267007 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:12.266969 5019 generic.go:356] "Generic (PLEG): container finished" podID="ab7c7403-6027-4cec-a6b7-82cca4bb8781" containerID="9dbe60b687417ffad7206457dbf6b4d0a5f0c91bf93eace35993244a0ab91d8f" exitCode=137 Nov 03 19:16:12.267206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:12.267016 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-996e09" Nov 03 19:16:12.267206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:12.267033 5019 scope.go:122] "RemoveContainer" containerID="9dbe60b687417ffad7206457dbf6b4d0a5f0c91bf93eace35993244a0ab91d8f" Nov 03 19:16:12.267449 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:12.267422 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8875-a22504" podUID="f9dc02d7-fdf4-4454-a315-977cb9abbecf" containerName="test-curl-8875-a22504" containerID="cri-o://89d87ac8d9c2c9d7059b61deb9f47e33bf2e97b6542fb64d1b5eaf4dc97d9191" gracePeriod=2 Nov 03 19:16:12.279276 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:12.279246 5019 scope.go:122] "RemoveContainer" containerID="9dbe60b687417ffad7206457dbf6b4d0a5f0c91bf93eace35993244a0ab91d8f" Nov 03 19:16:12.279598 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:12.279578 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbe60b687417ffad7206457dbf6b4d0a5f0c91bf93eace35993244a0ab91d8f\": container with ID starting with 9dbe60b687417ffad7206457dbf6b4d0a5f0c91bf93eace35993244a0ab91d8f not found: ID does not exist" containerID="9dbe60b687417ffad7206457dbf6b4d0a5f0c91bf93eace35993244a0ab91d8f" Nov 03 19:16:12.279683 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:12.279607 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbe60b687417ffad7206457dbf6b4d0a5f0c91bf93eace35993244a0ab91d8f"} err="failed to get container status \"9dbe60b687417ffad7206457dbf6b4d0a5f0c91bf93eace35993244a0ab91d8f\": rpc error: code = NotFound desc = could not find container \"9dbe60b687417ffad7206457dbf6b4d0a5f0c91bf93eace35993244a0ab91d8f\": container with ID starting with 9dbe60b687417ffad7206457dbf6b4d0a5f0c91bf93eace35993244a0ab91d8f not found: ID does not exist" Nov 03 19:16:12.949888 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:12.949856 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582"] Nov 03 19:16:12.950550 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:12.950535 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f9dc02d7-fdf4-4454-a315-977cb9abbecf" containerName="test-curl-8875-a22504" Nov 03 19:16:12.950597 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:12.950551 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9dc02d7-fdf4-4454-a315-977cb9abbecf" containerName="test-curl-8875-a22504" Nov 03 19:16:12.950700 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:12.950690 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9dc02d7-fdf4-4454-a315-977cb9abbecf" containerName="test-curl-8875-a22504" Nov 03 19:16:12.955191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:12.955166 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.006359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.005293 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3d4656-openshift-default-dockercfg-r7krk\"" Nov 03 19:16:13.229793 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.229649 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582"] Nov 03 19:16:13.300715 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.300675 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-podinfo\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.345561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.345501 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7knv\" (UniqueName: \"kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-kube-api-access-j7knv\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.345746 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.345619 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istiod-ca-cert\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.345746 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.345700 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-credential-socket\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.345868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.345748 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-token\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.345868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.345806 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-socket\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.345970 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.345863 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-certs\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.345970 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.345895 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-data\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.346076 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.345969 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-envoy\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.460032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.459989 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-podinfo\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.460312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.460278 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7knv\" (UniqueName: \"kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-kube-api-access-j7knv\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.464518 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.464481 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istiod-ca-cert\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.477701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.475379 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-credential-socket\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.477701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.475464 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-token\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.477701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.475517 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-socket\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.477701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.475585 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-certs\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.477701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.475612 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-data\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.477701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.475780 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-envoy\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.477701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.475988 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-podinfo\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.477701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.465978 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istiod-ca-cert\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.477701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.476178 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-socket\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.495837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.484737 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-data\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.495837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.484887 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-certs\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.495837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.485691 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-credential-socket\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.498264 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.498236 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-envoy\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.522198 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.522157 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7knv\" (UniqueName: \"kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-kube-api-access-j7knv\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.541475 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.541424 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-token\") pod \"e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:13.618437 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:13.618404 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:14.052312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.052285 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582"] Nov 03 19:16:14.052691 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:16:14.052545 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7cc1ef_c4de_4e81_9bbe_ad1c30db88da.slice/crio-1bd83a061e34e11346c90cadff7e5c389ecea8e9526b32d02abe791a6ce0293e WatchSource:0}: Error finding container 1bd83a061e34e11346c90cadff7e5c389ecea8e9526b32d02abe791a6ce0293e: Status 404 returned error can't find the container with id 1bd83a061e34e11346c90cadff7e5c389ecea8e9526b32d02abe791a6ce0293e Nov 03 19:16:14.056031 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.055996 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:16:14.056115 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.056061 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:16:14.056115 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.056090 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:16:14.383099 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.383061 5019 generic.go:356] "Generic (PLEG): container finished" podID="f9dc02d7-fdf4-4454-a315-977cb9abbecf" containerID="89d87ac8d9c2c9d7059b61deb9f47e33bf2e97b6542fb64d1b5eaf4dc97d9191" exitCode=137 Nov 03 19:16:14.405890 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.404544 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" event={"ID":"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da","Type":"ContainerStarted","Data":"6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989"} Nov 03 19:16:14.405890 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.404613 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" event={"ID":"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da","Type":"ContainerStarted","Data":"1bd83a061e34e11346c90cadff7e5c389ecea8e9526b32d02abe791a6ce0293e"} Nov 03 19:16:14.532074 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.523581 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" podStartSLOduration=2.523559543 podStartE2EDuration="2.523559543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:14.519376983 +0000 UTC m=+1908.491938163" watchObservedRunningTime="2026-11-03 19:16:14.523559543 +0000 UTC m=+1908.496120724" Nov 03 19:16:14.618615 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.618586 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:14.666348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.659290 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-a22504" Nov 03 19:16:14.784835 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.784805 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9dc02d7-fdf4-4454-a315-977cb9abbecf-kube-api-access-6qhd7\" (UniqueName: \"kubernetes.io/projected/f9dc02d7-fdf4-4454-a315-977cb9abbecf-kube-api-access-6qhd7\") pod \"f9dc02d7-fdf4-4454-a315-977cb9abbecf\" (UID: \"f9dc02d7-fdf4-4454-a315-977cb9abbecf\") " Nov 03 19:16:14.787014 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.786985 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9dc02d7-fdf4-4454-a315-977cb9abbecf-kube-api-access-6qhd7" pod "f9dc02d7-fdf4-4454-a315-977cb9abbecf" (UID: "f9dc02d7-fdf4-4454-a315-977cb9abbecf"). InnerVolumeSpecName "kube-api-access-6qhd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:14.886416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:14.886294 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qhd7\" (UniqueName: \"kubernetes.io/projected/f9dc02d7-fdf4-4454-a315-977cb9abbecf-kube-api-access-6qhd7\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:15.414626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:15.414588 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8875-a22504" Nov 03 19:16:15.415095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:15.414636 5019 scope.go:122] "RemoveContainer" containerID="89d87ac8d9c2c9d7059b61deb9f47e33bf2e97b6542fb64d1b5eaf4dc97d9191" Nov 03 19:16:15.471448 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:15.471408 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8875-a22504\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="f9dc02d7-fdf4-4454-a315-977cb9abbecf" pod="opendatahub/test-curl-8875-a22504" Nov 03 19:16:15.625302 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:15.625274 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:15.673047 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:15.673000 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8875-a22504\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="f9dc02d7-fdf4-4454-a315-977cb9abbecf" pod="opendatahub/test-curl-8875-a22504" Nov 03 19:16:16.154549 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:16.154472 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9dc02d7-fdf4-4454-a315-977cb9abbecf" path="/var/lib/kubelet/pods/f9dc02d7-fdf4-4454-a315-977cb9abbecf/volumes" Nov 03 19:16:16.197376 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:16.197303 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8875-a22504\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="f9dc02d7-fdf4-4454-a315-977cb9abbecf" pod="opendatahub/test-curl-8875-a22504" Nov 03 19:16:16.423867 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:16.423838 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:16.424765 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:16.424746 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:16:20.984873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:20.984822 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw"] Nov 03 19:16:20.998599 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:20.998470 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" Nov 03 19:16:21.059662 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.059622 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw"] Nov 03 19:16:21.087622 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.087581 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3d4656\"" Nov 03 19:16:21.088305 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.088193 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3d4656-dockercfg-h8tg4\"" Nov 03 19:16:21.126654 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.126534 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhlgf\" (UniqueName: \"kubernetes.io/projected/201aa634-3a94-4fb1-ab67-8f11817e6a7a-kube-api-access-rhlgf\") pod \"payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw\" (UID: \"201aa634-3a94-4fb1-ab67-8f11817e6a7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" Nov 03 19:16:21.126966 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.126943 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201aa634-3a94-4fb1-ab67-8f11817e6a7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw\" (UID: \"201aa634-3a94-4fb1-ab67-8f11817e6a7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" Nov 03 19:16:21.257619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.233823 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhlgf\" (UniqueName: \"kubernetes.io/projected/201aa634-3a94-4fb1-ab67-8f11817e6a7a-kube-api-access-rhlgf\") pod \"payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw\" (UID: \"201aa634-3a94-4fb1-ab67-8f11817e6a7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" Nov 03 19:16:21.286515 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.286485 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201aa634-3a94-4fb1-ab67-8f11817e6a7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw\" (UID: \"201aa634-3a94-4fb1-ab67-8f11817e6a7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" Nov 03 19:16:21.287387 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.286233 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk"] Nov 03 19:16:21.287841 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.287805 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201aa634-3a94-4fb1-ab67-8f11817e6a7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw\" (UID: \"201aa634-3a94-4fb1-ab67-8f11817e6a7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" Nov 03 19:16:21.301666 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.301639 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" Nov 03 19:16:21.302032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.301739 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk"] Nov 03 19:16:21.343906 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.343871 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhlgf\" (UniqueName: \"kubernetes.io/projected/201aa634-3a94-4fb1-ab67-8f11817e6a7a-kube-api-access-rhlgf\") pod \"payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw\" (UID: \"201aa634-3a94-4fb1-ab67-8f11817e6a7a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" Nov 03 19:16:21.387997 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.387962 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ededdd5-4be7-413c-8fb9-6be120d8119a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk\" (UID: \"1ededdd5-4be7-413c-8fb9-6be120d8119a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" Nov 03 19:16:21.388139 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.388018 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffpd\" (UniqueName: \"kubernetes.io/projected/1ededdd5-4be7-413c-8fb9-6be120d8119a-kube-api-access-2ffpd\") pod \"payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk\" (UID: \"1ededdd5-4be7-413c-8fb9-6be120d8119a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" Nov 03 19:16:21.489011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.488979 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ededdd5-4be7-413c-8fb9-6be120d8119a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk\" (UID: \"1ededdd5-4be7-413c-8fb9-6be120d8119a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" Nov 03 19:16:21.489167 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.489037 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffpd\" (UniqueName: \"kubernetes.io/projected/1ededdd5-4be7-413c-8fb9-6be120d8119a-kube-api-access-2ffpd\") pod \"payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk\" (UID: \"1ededdd5-4be7-413c-8fb9-6be120d8119a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" Nov 03 19:16:21.489757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.489735 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ededdd5-4be7-413c-8fb9-6be120d8119a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk\" (UID: \"1ededdd5-4be7-413c-8fb9-6be120d8119a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" Nov 03 19:16:21.546290 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.546226 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffpd\" (UniqueName: \"kubernetes.io/projected/1ededdd5-4be7-413c-8fb9-6be120d8119a-kube-api-access-2ffpd\") pod \"payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk\" (UID: \"1ededdd5-4be7-413c-8fb9-6be120d8119a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" Nov 03 19:16:21.623942 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.623914 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" Nov 03 19:16:21.626939 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:21.626914 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" Nov 03 19:16:22.004596 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:22.004564 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk"] Nov 03 19:16:22.005413 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:16:22.005381 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ededdd5_4be7_413c_8fb9_6be120d8119a.slice/crio-9f9cf64f86a0387e6d98a417ac0a018551cfecae1defe33f22aea4dfc90304ce WatchSource:0}: Error finding container 9f9cf64f86a0387e6d98a417ac0a018551cfecae1defe33f22aea4dfc90304ce: Status 404 returned error can't find the container with id 9f9cf64f86a0387e6d98a417ac0a018551cfecae1defe33f22aea4dfc90304ce Nov 03 19:16:22.051671 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:16:22.051648 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201aa634_3a94_4fb1_ab67_8f11817e6a7a.slice/crio-d000dc1996f9a3a41aadbccedb768117d90900dd0449946759f64479f7186227 WatchSource:0}: Error finding container d000dc1996f9a3a41aadbccedb768117d90900dd0449946759f64479f7186227: Status 404 returned error can't find the container with id d000dc1996f9a3a41aadbccedb768117d90900dd0449946759f64479f7186227 Nov 03 19:16:22.051785 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:22.051724 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw"] Nov 03 19:16:22.473155 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:22.473119 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" event={"ID":"201aa634-3a94-4fb1-ab67-8f11817e6a7a","Type":"ContainerStarted","Data":"5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f"} Nov 03 19:16:22.473362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:22.473170 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" event={"ID":"201aa634-3a94-4fb1-ab67-8f11817e6a7a","Type":"ContainerStarted","Data":"d000dc1996f9a3a41aadbccedb768117d90900dd0449946759f64479f7186227"} Nov 03 19:16:22.473362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:22.473305 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" Nov 03 19:16:22.474823 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:22.474798 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" event={"ID":"1ededdd5-4be7-413c-8fb9-6be120d8119a","Type":"ContainerStarted","Data":"9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c"} Nov 03 19:16:22.474920 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:22.474841 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" event={"ID":"1ededdd5-4be7-413c-8fb9-6be120d8119a","Type":"ContainerStarted","Data":"9f9cf64f86a0387e6d98a417ac0a018551cfecae1defe33f22aea4dfc90304ce"} Nov 03 19:16:22.474920 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:22.474867 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" Nov 03 19:16:22.582733 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:22.582670 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" podStartSLOduration=2.5826576980000002 podStartE2EDuration="2.582657698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:22.581745038 +0000 UTC m=+1916.554306220" watchObservedRunningTime="2026-11-03 19:16:22.582657698 +0000 UTC m=+1916.555218879" Nov 03 19:16:22.698348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:22.698269 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" podStartSLOduration=1.698168315 podStartE2EDuration="1.698168315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:22.690878382 +0000 UTC m=+1916.663439564" watchObservedRunningTime="2026-11-03 19:16:22.698168315 +0000 UTC m=+1916.670729498" Nov 03 19:16:29.110900 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.110863 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4"] Nov 03 19:16:29.116839 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.116820 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.165658 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.165631 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-9dcd6c-serving-cert\"" Nov 03 19:16:29.172803 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.172778 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4"] Nov 03 19:16:29.225272 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.225239 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.225506 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.225296 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llg89\" (UniqueName: \"kubernetes.io/projected/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-kube-api-access-llg89\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.225586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.225503 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.225586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.225548 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.326964 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.326923 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.327166 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.326995 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llg89\" (UniqueName: \"kubernetes.io/projected/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-kube-api-access-llg89\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.327166 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.327091 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.327166 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:29.327094 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-serving-cert: secret "maas-api-e2e-shared-a-w3-9dcd6c-serving-cert" not found Nov 03 19:16:29.327166 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.327117 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.327166 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:29.327167 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-maas-api-tls podName:b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b nodeName:}" failed. No retries permitted until 2026-11-03 19:16:29.827147832 +0000 UTC m=+1923.799709007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-maas-api-tls") pod "maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" (UID: "b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b") : secret "maas-api-e2e-shared-a-w3-9dcd6c-serving-cert" not found Nov 03 19:16:29.327779 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.327752 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.329727 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.329704 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.392435 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.389400 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llg89\" (UniqueName: \"kubernetes.io/projected/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-kube-api-access-llg89\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.851303 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.851267 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:29.853868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:29.853838 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:30.031249 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:30.031215 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:30.410484 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:16:30.409774 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99aaf9c_4f4e_4e1d_a1aa_a6b9c4f1b12b.slice/crio-b6ea6bbab12d4bc184c706127ff06334d73602759cba9983b500c735c6dd642d WatchSource:0}: Error finding container b6ea6bbab12d4bc184c706127ff06334d73602759cba9983b500c735c6dd642d: Status 404 returned error can't find the container with id b6ea6bbab12d4bc184c706127ff06334d73602759cba9983b500c735c6dd642d Nov 03 19:16:30.411837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:30.411808 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4"] Nov 03 19:16:30.637530 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:30.637497 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" event={"ID":"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b","Type":"ContainerStarted","Data":"b6ea6bbab12d4bc184c706127ff06334d73602759cba9983b500c735c6dd642d"} Nov 03 19:16:31.647712 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:31.647574 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" event={"ID":"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b","Type":"ContainerStarted","Data":"9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b"} Nov 03 19:16:31.648158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:31.647797 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:31.758889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:31.758834 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" podStartSLOduration=1.875989515 podStartE2EDuration="2.758816275s" totalImagesPullingTime="882.82676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:31.758402701 +0000 UTC m=+1925.730963884" watchObservedRunningTime="2026-11-03 19:16:31.758816275 +0000 UTC m=+1925.731377459" Nov 03 19:16:33.485747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:33.485718 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" Nov 03 19:16:33.486214 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:33.485838 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" Nov 03 19:16:35.740780 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.740696 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt"] Nov 03 19:16:35.746071 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.746049 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:35.795397 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.795367 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1cc3f2-openshift-default-dockercfg-smhsf\"" Nov 03 19:16:35.849838 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.849346 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt"] Nov 03 19:16:35.913438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.895436 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm8vc\" (UniqueName: \"kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-kube-api-access-dm8vc\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:35.913438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.895503 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-data\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:35.913438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.895543 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-token\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:35.913438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.895678 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96db2630-ed5b-49f8-b716-53ffd18dfa33-istiod-ca-cert\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:35.913438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.895724 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-envoy\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:35.913438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.895749 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-socket\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:35.913438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.895939 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-certs\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:35.913438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.896009 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-credential-socket\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:35.913438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:35.896062 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-podinfo\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.003008 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.001962 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm8vc\" (UniqueName: \"kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-kube-api-access-dm8vc\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.003184 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.003015 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-data\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.003184 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.003066 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-token\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.003300 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.003188 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96db2630-ed5b-49f8-b716-53ffd18dfa33-istiod-ca-cert\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.003300 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.003229 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-envoy\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.003300 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.003254 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-socket\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.003480 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.003431 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-certs\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.003533 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.003487 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-credential-socket\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.003588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.003537 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-podinfo\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.023816 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.004981 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-data\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.023816 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.005578 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-credential-socket\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.023816 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.005823 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-socket\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.023816 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.006500 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96db2630-ed5b-49f8-b716-53ffd18dfa33-istiod-ca-cert\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.023816 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.006593 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-certs\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.046011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.044878 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-envoy\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.046011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.045628 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-podinfo\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.068466 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.067605 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-token\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.071491 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.071259 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm8vc\" (UniqueName: \"kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-kube-api-access-dm8vc\") pod \"e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.362811 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.362716 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:36.788973 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:16:36.788928 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96db2630_ed5b_49f8_b716_53ffd18dfa33.slice/crio-99fb9f22ea4d80ca3e960d353079b97f3d700d7598d72c61599f24048fb286ae WatchSource:0}: Error finding container 99fb9f22ea4d80ca3e960d353079b97f3d700d7598d72c61599f24048fb286ae: Status 404 returned error can't find the container with id 99fb9f22ea4d80ca3e960d353079b97f3d700d7598d72c61599f24048fb286ae Nov 03 19:16:36.789458 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.789099 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt"] Nov 03 19:16:36.792243 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.792201 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:16:36.792314 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.792271 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:16:36.792314 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:36.792302 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:16:37.672487 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:37.672458 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:16:37.718725 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:37.718692 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" event={"ID":"96db2630-ed5b-49f8-b716-53ffd18dfa33","Type":"ContainerStarted","Data":"e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c"} Nov 03 19:16:37.718892 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:37.718751 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" event={"ID":"96db2630-ed5b-49f8-b716-53ffd18dfa33","Type":"ContainerStarted","Data":"99fb9f22ea4d80ca3e960d353079b97f3d700d7598d72c61599f24048fb286ae"} Nov 03 19:16:37.898260 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:37.896760 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" podStartSLOduration=2.896741737 podStartE2EDuration="2.896741737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:37.894834341 +0000 UTC m=+1931.867395554" watchObservedRunningTime="2026-11-03 19:16:37.896741737 +0000 UTC m=+1931.869302922" Nov 03 19:16:38.363472 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:38.363432 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:38.370130 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:38.370101 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:38.726686 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:38.726653 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:38.727850 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:38.727827 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:16:41.299198 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:41.299165 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:16:44.333351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.332824 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk"] Nov 03 19:16:44.355351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.354034 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" Nov 03 19:16:44.382386 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.381533 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x"] Nov 03 19:16:44.400349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.389498 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk"] Nov 03 19:16:44.400349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.389624 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:44.410006 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.409968 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdznk\" (UniqueName: \"kubernetes.io/projected/7373d851-49e8-4249-bd94-2a5c95e3d625-kube-api-access-mdznk\") pod \"payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk\" (UID: \"7373d851-49e8-4249-bd94-2a5c95e3d625\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" Nov 03 19:16:44.410390 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.410367 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7373d851-49e8-4249-bd94-2a5c95e3d625-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk\" (UID: \"7373d851-49e8-4249-bd94-2a5c95e3d625\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" Nov 03 19:16:44.433709 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.433664 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1cc3f2\"" Nov 03 19:16:44.434478 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.434432 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1cc3f2-dockercfg-pfknb\"" Nov 03 19:16:44.477347 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.447736 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x"] Nov 03 19:16:44.526351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.522777 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-4f1a90-serving-cert\"" Nov 03 19:16:44.526351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.523829 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdznk\" (UniqueName: \"kubernetes.io/projected/7373d851-49e8-4249-bd94-2a5c95e3d625-kube-api-access-mdznk\") pod \"payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk\" (UID: \"7373d851-49e8-4249-bd94-2a5c95e3d625\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" Nov 03 19:16:44.526351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.523887 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0a76ec-c204-473d-8b44-1bb35920dceb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:44.526351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.524011 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:44.526351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.524079 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7373d851-49e8-4249-bd94-2a5c95e3d625-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk\" (UID: \"7373d851-49e8-4249-bd94-2a5c95e3d625\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" Nov 03 19:16:44.526351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.524131 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:44.526351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.524183 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dph8t\" (UniqueName: \"kubernetes.io/projected/5c0a76ec-c204-473d-8b44-1bb35920dceb-kube-api-access-dph8t\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:44.526351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.525622 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7373d851-49e8-4249-bd94-2a5c95e3d625-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk\" (UID: \"7373d851-49e8-4249-bd94-2a5c95e3d625\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" Nov 03 19:16:44.643413 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.643307 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdznk\" (UniqueName: \"kubernetes.io/projected/7373d851-49e8-4249-bd94-2a5c95e3d625-kube-api-access-mdznk\") pod \"payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk\" (UID: \"7373d851-49e8-4249-bd94-2a5c95e3d625\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" Nov 03 19:16:44.661578 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.661533 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:44.661918 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.661901 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:44.662061 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.662048 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dph8t\" (UniqueName: \"kubernetes.io/projected/5c0a76ec-c204-473d-8b44-1bb35920dceb-kube-api-access-dph8t\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:44.666068 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.666036 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0a76ec-c204-473d-8b44-1bb35920dceb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:44.667840 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.667809 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0a76ec-c204-473d-8b44-1bb35920dceb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:44.698349 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:44.668110 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-serving-cert: secret "maas-api-e2e-shared-b-w2-4f1a90-serving-cert" not found Nov 03 19:16:44.698349 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:44.690072 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-maas-api-tls podName:5c0a76ec-c204-473d-8b44-1bb35920dceb nodeName:}" failed. No retries permitted until 2026-11-03 19:16:45.189978396 +0000 UTC m=+1939.162539557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-maas-api-tls") pod "maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" (UID: "5c0a76ec-c204-473d-8b44-1bb35920dceb") : secret "maas-api-e2e-shared-b-w2-4f1a90-serving-cert" not found Nov 03 19:16:44.698349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.695633 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:44.747245 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.743802 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dph8t\" (UniqueName: \"kubernetes.io/projected/5c0a76ec-c204-473d-8b44-1bb35920dceb-kube-api-access-dph8t\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:44.747245 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:44.744312 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" Nov 03 19:16:45.195436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:45.195306 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:45.198247 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:45.198211 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:45.238972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:45.238941 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:45.267372 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:16:45.261946 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7373d851_49e8_4249_bd94_2a5c95e3d625.slice/crio-15fb68c90f32ed338d8c944b4767817f12a5ea5bed6158aefec57fa88c8608bf WatchSource:0}: Error finding container 15fb68c90f32ed338d8c944b4767817f12a5ea5bed6158aefec57fa88c8608bf: Status 404 returned error can't find the container with id 15fb68c90f32ed338d8c944b4767817f12a5ea5bed6158aefec57fa88c8608bf Nov 03 19:16:45.267372 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:45.261977 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk"] Nov 03 19:16:45.792529 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:45.792435 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" event={"ID":"7373d851-49e8-4249-bd94-2a5c95e3d625","Type":"ContainerStarted","Data":"8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37"} Nov 03 19:16:45.792529 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:45.792500 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" event={"ID":"7373d851-49e8-4249-bd94-2a5c95e3d625","Type":"ContainerStarted","Data":"15fb68c90f32ed338d8c944b4767817f12a5ea5bed6158aefec57fa88c8608bf"} Nov 03 19:16:45.792529 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:45.792528 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" Nov 03 19:16:45.839270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:45.839241 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x"] Nov 03 19:16:45.839454 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:16:45.839418 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0a76ec_c204_473d_8b44_1bb35920dceb.slice/crio-92a89a5bd6046f0cb88647ccd6c003ca42fecfdc2ff56528cad023a25cd91aad WatchSource:0}: Error finding container 92a89a5bd6046f0cb88647ccd6c003ca42fecfdc2ff56528cad023a25cd91aad: Status 404 returned error can't find the container with id 92a89a5bd6046f0cb88647ccd6c003ca42fecfdc2ff56528cad023a25cd91aad Nov 03 19:16:45.901764 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:45.901705 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" podStartSLOduration=1.901685535 podStartE2EDuration="1.901685535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:45.900723181 +0000 UTC m=+1939.873284386" watchObservedRunningTime="2026-11-03 19:16:45.901685535 +0000 UTC m=+1939.874246721" Nov 03 19:16:46.802080 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:46.802045 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" event={"ID":"5c0a76ec-c204-473d-8b44-1bb35920dceb","Type":"ContainerStarted","Data":"92a89a5bd6046f0cb88647ccd6c003ca42fecfdc2ff56528cad023a25cd91aad"} Nov 03 19:16:47.814004 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:47.813966 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" event={"ID":"5c0a76ec-c204-473d-8b44-1bb35920dceb","Type":"ContainerStarted","Data":"7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6"} Nov 03 19:16:47.814531 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:47.814034 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:47.933374 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:47.933309 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" podStartSLOduration=3.06900312 podStartE2EDuration="3.933296784s" totalImagesPullingTime="864.293664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:47.930360789 +0000 UTC m=+1941.902921971" watchObservedRunningTime="2026-11-03 19:16:47.933296784 +0000 UTC m=+1941.905857966" Nov 03 19:16:53.851212 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:53.851181 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:16:56.350415 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.350376 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8863-f6d127"] Nov 03 19:16:56.354849 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.354829 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8863-f6d127" Nov 03 19:16:56.405385 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.405354 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2f8z4\"" Nov 03 19:16:56.411617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.410633 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8863-f6d127"] Nov 03 19:16:56.415891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.415860 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjnjz\" (UniqueName: \"kubernetes.io/projected/8b9904bd-e0fd-49a1-a353-802771af0744-kube-api-access-gjnjz\") pod \"test-curl-8863-f6d127\" (UID: \"8b9904bd-e0fd-49a1-a353-802771af0744\") " pod="opendatahub/test-curl-8863-f6d127" Nov 03 19:16:56.517493 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.517405 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjnjz\" (UniqueName: \"kubernetes.io/projected/8b9904bd-e0fd-49a1-a353-802771af0744-kube-api-access-gjnjz\") pod \"test-curl-8863-f6d127\" (UID: \"8b9904bd-e0fd-49a1-a353-802771af0744\") " pod="opendatahub/test-curl-8863-f6d127" Nov 03 19:16:56.575200 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.575169 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjnjz\" (UniqueName: \"kubernetes.io/projected/8b9904bd-e0fd-49a1-a353-802771af0744-kube-api-access-gjnjz\") pod \"test-curl-8863-f6d127\" (UID: \"8b9904bd-e0fd-49a1-a353-802771af0744\") " pod="opendatahub/test-curl-8863-f6d127" Nov 03 19:16:56.715395 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.715366 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8863-f6d127" Nov 03 19:16:56.810028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.809995 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" Nov 03 19:16:56.924448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.924413 5019 generic.go:356] "Generic (PLEG): container finished" podID="641da692-25ae-4e05-9e06-2417ab73918d" containerID="e59c1a0755e146175e8717f9766142ea60347220ffec2358f039f5b6c323445f" exitCode=0 Nov 03 19:16:56.924448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.924443 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" event={"ID":"641da692-25ae-4e05-9e06-2417ab73918d","Type":"ContainerDied","Data":"e59c1a0755e146175e8717f9766142ea60347220ffec2358f039f5b6c323445f"} Nov 03 19:16:56.924799 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.924784 5019 scope.go:122] "RemoveContainer" containerID="e59c1a0755e146175e8717f9766142ea60347220ffec2358f039f5b6c323445f" Nov 03 19:16:56.926401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.926374 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:16:56.926475 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.926438 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:16:56.926512 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:56.926471 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:16:57.133548 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:57.131572 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8863-f6d127"] Nov 03 19:16:57.933783 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:57.933747 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" event={"ID":"641da692-25ae-4e05-9e06-2417ab73918d","Type":"ContainerStarted","Data":"1778f4fc9111809980b7cefada33d69d25ff46b80ae79cfef3ebd8bae41ce580"} Nov 03 19:16:57.935659 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:57.935628 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8863-f6d127" event={"ID":"8b9904bd-e0fd-49a1-a353-802771af0744","Type":"ContainerStarted","Data":"ed7e505c0202ac758b1c87415f98053b21b05178091fb8d5c5229db4cb4af727"} Nov 03 19:16:57.935993 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:57.935671 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8863-f6d127" event={"ID":"8b9904bd-e0fd-49a1-a353-802771af0744","Type":"ContainerStarted","Data":"303629024c9d86dfb2725d90e9d461693d842b2894549bb568dadd9c86a28bec"} Nov 03 19:16:58.154173 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:58.154119 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8863-f6d127" podStartSLOduration=2.154106463 podStartE2EDuration="2.154106463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:58.154008533 +0000 UTC m=+1952.126569714" watchObservedRunningTime="2026-11-03 19:16:58.154106463 +0000 UTC m=+1952.126667645" Nov 03 19:16:58.491921 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:58.491890 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:16:58.492276 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:58.492193 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:16:58.492448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:58.492425 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" podUID="641da692-25ae-4e05-9e06-2417ab73918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:16:59.495785 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:59.495743 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:16:59.496302 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:59.495804 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" podUID="641da692-25ae-4e05-9e06-2417ab73918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:16:59.721655 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:59.721619 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8863-f6d127"] Nov 03 19:16:59.725544 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:59.725517 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8863-f6d127"] Nov 03 19:16:59.950525 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:16:59.950486 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8863-f6d127" podUID="8b9904bd-e0fd-49a1-a353-802771af0744" containerName="test-curl-8863-f6d127" containerID="cri-o://ed7e505c0202ac758b1c87415f98053b21b05178091fb8d5c5229db4cb4af727" gracePeriod=2 Nov 03 19:16:59.998542 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:16:59.998502 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8863-f6d127\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="8b9904bd-e0fd-49a1-a353-802771af0744" pod="opendatahub/test-curl-8863-f6d127" Nov 03 19:17:00.492352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:00.488726 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:17:00.492352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:00.488789 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" podUID="641da692-25ae-4e05-9e06-2417ab73918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:17:01.489050 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:01.489009 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:17:01.489543 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:01.489082 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" podUID="641da692-25ae-4e05-9e06-2417ab73918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:17:02.239995 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:02.239967 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8863-f6d127" Nov 03 19:17:02.351513 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:02.351474 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9904bd-e0fd-49a1-a353-802771af0744-kube-api-access-gjnjz\" (UniqueName: \"kubernetes.io/projected/8b9904bd-e0fd-49a1-a353-802771af0744-kube-api-access-gjnjz\") pod \"8b9904bd-e0fd-49a1-a353-802771af0744\" (UID: \"8b9904bd-e0fd-49a1-a353-802771af0744\") " Nov 03 19:17:02.353750 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:02.353722 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9904bd-e0fd-49a1-a353-802771af0744-kube-api-access-gjnjz" pod "8b9904bd-e0fd-49a1-a353-802771af0744" (UID: "8b9904bd-e0fd-49a1-a353-802771af0744"). InnerVolumeSpecName "kube-api-access-gjnjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:02.453117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:02.453079 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjnjz\" (UniqueName: \"kubernetes.io/projected/8b9904bd-e0fd-49a1-a353-802771af0744-kube-api-access-gjnjz\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:02.489261 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:02.489170 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:17:02.489261 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:02.489240 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" podUID="641da692-25ae-4e05-9e06-2417ab73918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:17:02.578777 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:02.576427 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:17:02.999633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:02.999596 5019 generic.go:356] "Generic (PLEG): container finished" podID="8b9904bd-e0fd-49a1-a353-802771af0744" containerID="ed7e505c0202ac758b1c87415f98053b21b05178091fb8d5c5229db4cb4af727" exitCode=137 Nov 03 19:17:02.999830 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:02.999639 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8863-f6d127" Nov 03 19:17:02.999830 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:02.999669 5019 scope.go:122] "RemoveContainer" containerID="ed7e505c0202ac758b1c87415f98053b21b05178091fb8d5c5229db4cb4af727" Nov 03 19:17:03.011156 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:03.011135 5019 scope.go:122] "RemoveContainer" containerID="ed7e505c0202ac758b1c87415f98053b21b05178091fb8d5c5229db4cb4af727" Nov 03 19:17:03.011496 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:17:03.011474 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7e505c0202ac758b1c87415f98053b21b05178091fb8d5c5229db4cb4af727\": container with ID starting with ed7e505c0202ac758b1c87415f98053b21b05178091fb8d5c5229db4cb4af727 not found: ID does not exist" containerID="ed7e505c0202ac758b1c87415f98053b21b05178091fb8d5c5229db4cb4af727" Nov 03 19:17:03.011563 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:03.011505 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7e505c0202ac758b1c87415f98053b21b05178091fb8d5c5229db4cb4af727"} err="failed to get container status \"ed7e505c0202ac758b1c87415f98053b21b05178091fb8d5c5229db4cb4af727\": rpc error: code = NotFound desc = could not find container \"ed7e505c0202ac758b1c87415f98053b21b05178091fb8d5c5229db4cb4af727\": container with ID starting with ed7e505c0202ac758b1c87415f98053b21b05178091fb8d5c5229db4cb4af727 not found: ID does not exist" Nov 03 19:17:03.056757 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:17:03.056714 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8863-f6d127\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="8b9904bd-e0fd-49a1-a353-802771af0744" pod="opendatahub/test-curl-8863-f6d127" Nov 03 19:17:03.493663 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:03.493632 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:17:03.494574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:03.494555 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:17:03.541534 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:17:03.541489 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8863-f6d127\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="8b9904bd-e0fd-49a1-a353-802771af0744" pod="opendatahub/test-curl-8863-f6d127" Nov 03 19:17:03.748467 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:17:03.748384 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8863-f6d127\" is forbidden: User \"system:node:ip-10-0-139-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-179.us-west-2.compute.internal' and this object" podUID="8b9904bd-e0fd-49a1-a353-802771af0744" pod="opendatahub/test-curl-8863-f6d127" Nov 03 19:17:04.152466 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:04.152374 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b9904bd-e0fd-49a1-a353-802771af0744" path="/var/lib/kubelet/pods/8b9904bd-e0fd-49a1-a353-802771af0744/volumes" Nov 03 19:17:08.342695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.342664 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt"] Nov 03 19:17:08.344053 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.344026 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8b9904bd-e0fd-49a1-a353-802771af0744" containerName="test-curl-8863-f6d127" Nov 03 19:17:08.344053 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.344049 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9904bd-e0fd-49a1-a353-802771af0744" containerName="test-curl-8863-f6d127" Nov 03 19:17:08.344209 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.344167 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9904bd-e0fd-49a1-a353-802771af0744" containerName="test-curl-8863-f6d127" Nov 03 19:17:08.349364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.349306 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:08.371676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.371650 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-maas-api-tls\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:08.371887 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.371871 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s5vd\" (UniqueName: \"kubernetes.io/projected/a050632f-9924-48ed-8271-ed06f65445a6-kube-api-access-9s5vd\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:08.371937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.371925 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a050632f-9924-48ed-8271-ed06f65445a6-openshift-service-ca\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:08.372035 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.372012 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-metrics-certs\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:08.448572 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.448538 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-dd1327-serving-cert\"" Nov 03 19:17:08.452960 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.452931 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt"] Nov 03 19:17:08.473125 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.473091 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s5vd\" (UniqueName: \"kubernetes.io/projected/a050632f-9924-48ed-8271-ed06f65445a6-kube-api-access-9s5vd\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:08.473312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.473178 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a050632f-9924-48ed-8271-ed06f65445a6-openshift-service-ca\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:08.473312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.473228 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-metrics-certs\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:08.473312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.473305 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-maas-api-tls\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:08.473562 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:17:08.473543 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-serving-cert: secret "maas-api-e2e-api-b-dd1327-serving-cert" not found Nov 03 19:17:08.474451 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:17:08.474433 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-maas-api-tls podName:a050632f-9924-48ed-8271-ed06f65445a6 nodeName:}" failed. No retries permitted until 2026-11-03 19:17:08.974410007 +0000 UTC m=+1962.946971167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-maas-api-tls") pod "maas-api-e2e-api-b-dd1327-6666596548-tq4nt" (UID: "a050632f-9924-48ed-8271-ed06f65445a6") : secret "maas-api-e2e-api-b-dd1327-serving-cert" not found Nov 03 19:17:08.475129 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.475106 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a050632f-9924-48ed-8271-ed06f65445a6-openshift-service-ca\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:08.476845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.476822 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-metrics-certs\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:08.531493 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:08.531466 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s5vd\" (UniqueName: \"kubernetes.io/projected/a050632f-9924-48ed-8271-ed06f65445a6-kube-api-access-9s5vd\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:09.009720 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:09.009679 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-maas-api-tls\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:09.012227 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:09.012199 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-maas-api-tls\") pod \"maas-api-e2e-api-b-dd1327-6666596548-tq4nt\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:09.263677 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:09.263598 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:09.648423 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:17:09.648391 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda050632f_9924_48ed_8271_ed06f65445a6.slice/crio-f86c06bb69135043162886a5c0f82b151096eacc9c7007eaba4032826c3d225f WatchSource:0}: Error finding container f86c06bb69135043162886a5c0f82b151096eacc9c7007eaba4032826c3d225f: Status 404 returned error can't find the container with id f86c06bb69135043162886a5c0f82b151096eacc9c7007eaba4032826c3d225f Nov 03 19:17:09.650508 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:09.650476 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt"] Nov 03 19:17:10.052621 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:10.052585 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" event={"ID":"a050632f-9924-48ed-8271-ed06f65445a6","Type":"ContainerStarted","Data":"f86c06bb69135043162886a5c0f82b151096eacc9c7007eaba4032826c3d225f"} Nov 03 19:17:11.061260 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:11.061211 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" event={"ID":"a050632f-9924-48ed-8271-ed06f65445a6","Type":"ContainerStarted","Data":"e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b"} Nov 03 19:17:11.061781 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:11.061283 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:11.174267 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:11.174214 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" podStartSLOduration=2.305486095 podStartE2EDuration="3.174199744s" totalImagesPullingTime="868.713649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:17:11.171351098 +0000 UTC m=+1965.143912294" watchObservedRunningTime="2026-11-03 19:17:11.174199744 +0000 UTC m=+1965.146760926" Nov 03 19:17:11.329880 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:17:11.329780 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:17:17.075809 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:17.075776 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:17:32.450486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:32.450450 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj"] Nov 03 19:17:32.451037 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:32.450656 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerName="istio-proxy" containerID="cri-o://3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21" gracePeriod=30 Nov 03 19:17:35.904574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:35.904519 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:17:35.905239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:35.905188 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:17:37.842537 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.842506 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:17:37.981570 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.981530 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-kube-api-access-h8qtf\" (UniqueName: \"kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-kube-api-access-h8qtf\") pod \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " Nov 03 19:17:37.981769 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.981600 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-credential-socket\") pod \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " Nov 03 19:17:37.981769 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.981725 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-podinfo\") pod \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " Nov 03 19:17:37.981889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.981770 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-envoy\") pod \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " Nov 03 19:17:37.981889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.981829 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-socket\") pod \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " Nov 03 19:17:37.982010 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.981928 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istiod-ca-cert\") pod \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " Nov 03 19:17:37.982066 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.982009 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-token\" (UniqueName: \"kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-token\") pod \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " Nov 03 19:17:37.982066 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.982044 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-data\") pod \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " Nov 03 19:17:37.982169 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.982138 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-certs\") pod \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\" (UID: \"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd\") " Nov 03 19:17:37.982435 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.982406 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-socket" pod "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" (UID: "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:37.982533 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.982485 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istiod-ca-cert" pod "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" (UID: "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:17:37.982908 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.982807 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-data" pod "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" (UID: "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:37.982908 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.982884 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-certs" pod "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" (UID: "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:37.983154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.982935 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:37.983154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.982958 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:37.983154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.983024 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-credential-socket" pod "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" (UID: "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:37.984683 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.984645 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-envoy" pod "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" (UID: "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:37.985199 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.985168 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-kube-api-access-h8qtf" pod "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" (UID: "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd"). InnerVolumeSpecName "kube-api-access-h8qtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:37.986272 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.986173 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-token" pod "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" (UID: "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:37.986726 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:37.986702 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-podinfo" pod "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" (UID: "748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:17:38.084708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.084498 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.084708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.084533 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.084708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.084548 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.084708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.084563 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8qtf\" (UniqueName: \"kubernetes.io/projected/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-kube-api-access-h8qtf\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.084708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.084579 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.084708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.084594 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.084708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.084607 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:38.471546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.471509 5019 generic.go:356] "Generic (PLEG): container finished" podID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerID="3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21" exitCode=0 Nov 03 19:17:38.471776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.471574 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" Nov 03 19:17:38.471776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.471602 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" event={"ID":"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd","Type":"ContainerDied","Data":"3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21"} Nov 03 19:17:38.471776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.471650 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj" event={"ID":"748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd","Type":"ContainerDied","Data":"22c073fed15922fccbc65dfa2e3b1c5e6e5390072ec908ff678a008b4f63c321"} Nov 03 19:17:38.471776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.471679 5019 scope.go:122] "RemoveContainer" containerID="3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21" Nov 03 19:17:38.484901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.484874 5019 scope.go:122] "RemoveContainer" containerID="3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21" Nov 03 19:17:38.485261 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:17:38.485228 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21\": container with ID starting with 3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21 not found: ID does not exist" containerID="3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21" Nov 03 19:17:38.485393 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.485273 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21"} err="failed to get container status \"3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21\": rpc error: code = NotFound desc = could not find container \"3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21\": container with ID starting with 3c778062c51f549c9069c80c5ac2c1077b20c96bff3fe70881ff2e6623635e21 not found: ID does not exist" Nov 03 19:17:38.640928 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.640861 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj"] Nov 03 19:17:38.647798 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:38.647715 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-t5tjj"] Nov 03 19:17:39.666599 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:39.666556 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:17:39.667156 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:39.667124 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:17:40.178362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:40.174854 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" path="/var/lib/kubelet/pods/748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd/volumes" Nov 03 19:17:41.301409 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:17:41.299826 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:17:41.326568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.326540 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5"] Nov 03 19:17:41.327404 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.327382 5019 cpu_manager.go:414] "containerMap: removing container" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerName="istio-proxy" Nov 03 19:17:41.327404 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.327400 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerName="istio-proxy" Nov 03 19:17:41.327562 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.327543 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="748bcc6f-9eed-4a21-acdc-3c9cbdbdf4dd" containerName="istio-proxy" Nov 03 19:17:41.331216 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.331192 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:41.380131 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.380090 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3d4656-serving-cert\"" Nov 03 19:17:41.434991 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.434945 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5"] Nov 03 19:17:41.462475 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.462443 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604159f6-5047-43b8-b760-8083a8f2ef3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:41.462740 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.462635 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:41.462740 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.462665 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-metrics-certs\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:41.462740 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.462699 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654qn\" (UniqueName: \"kubernetes.io/projected/604159f6-5047-43b8-b760-8083a8f2ef3c-kube-api-access-654qn\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:41.515343 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.515304 5019 generic.go:356] "Generic (PLEG): container finished" podID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerID="3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de" exitCode=0 Nov 03 19:17:41.515506 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.515382 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" event={"ID":"ad088e1c-645f-4365-89ac-6e8f85589ddb","Type":"ContainerDied","Data":"3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de"} Nov 03 19:17:41.515934 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.515914 5019 scope.go:122] "RemoveContainer" containerID="3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de" Nov 03 19:17:41.517803 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.517773 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:17:41.517897 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.517845 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:17:41.517897 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.517876 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:17:41.604362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.598054 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604159f6-5047-43b8-b760-8083a8f2ef3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:41.604362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.598223 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:41.604362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.598255 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-metrics-certs\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:41.604362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.598292 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654qn\" (UniqueName: \"kubernetes.io/projected/604159f6-5047-43b8-b760-8083a8f2ef3c-kube-api-access-654qn\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:41.618352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.616408 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604159f6-5047-43b8-b760-8083a8f2ef3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:41.618352 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:17:41.616554 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-serving-cert: secret "maas-api-e2e-inf-a-3d4656-serving-cert" not found Nov 03 19:17:41.618352 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:17:41.616619 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-maas-api-tls podName:604159f6-5047-43b8-b760-8083a8f2ef3c nodeName:}" failed. No retries permitted until 2026-11-03 19:17:42.116597791 +0000 UTC m=+1996.089158966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-maas-api-tls") pod "maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" (UID: "604159f6-5047-43b8-b760-8083a8f2ef3c") : secret "maas-api-e2e-inf-a-3d4656-serving-cert" not found Nov 03 19:17:41.661428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.654949 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-metrics-certs\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:41.691552 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:41.691475 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-654qn\" (UniqueName: \"kubernetes.io/projected/604159f6-5047-43b8-b760-8083a8f2ef3c-kube-api-access-654qn\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:42.216271 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.216184 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:42.218645 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.218617 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:42.245545 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.245518 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:42.526892 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.526862 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" event={"ID":"ad088e1c-645f-4365-89ac-6e8f85589ddb","Type":"ContainerStarted","Data":"01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3"} Nov 03 19:17:42.528499 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.528474 5019 generic.go:356] "Generic (PLEG): container finished" podID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerID="77319ec8bfb0a30b0c793c586cae4beea50580b908a35be3482660502ada0ec1" exitCode=0 Nov 03 19:17:42.528652 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.528544 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" event={"ID":"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996","Type":"ContainerDied","Data":"77319ec8bfb0a30b0c793c586cae4beea50580b908a35be3482660502ada0ec1"} Nov 03 19:17:42.528894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.528879 5019 scope.go:122] "RemoveContainer" containerID="77319ec8bfb0a30b0c793c586cae4beea50580b908a35be3482660502ada0ec1" Nov 03 19:17:42.530422 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.530392 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:17:42.530510 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.530461 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:17:42.530510 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.530498 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:17:42.632094 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.632021 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5"] Nov 03 19:17:42.632260 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:17:42.632244 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604159f6_5047_43b8_b760_8083a8f2ef3c.slice/crio-f32b9c3e26d34e95e44ace8c1a43a30fcc463d04d5bd1b9d40f0ff8cb91a63fa WatchSource:0}: Error finding container f32b9c3e26d34e95e44ace8c1a43a30fcc463d04d5bd1b9d40f0ff8cb91a63fa: Status 404 returned error can't find the container with id f32b9c3e26d34e95e44ace8c1a43a30fcc463d04d5bd1b9d40f0ff8cb91a63fa Nov 03 19:17:42.688271 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.686953 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:17:42.688271 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.687070 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:17:42.688271 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:42.687113 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:17:43.538870 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:43.538827 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" event={"ID":"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996","Type":"ContainerStarted","Data":"cefe8380de958fefa16f69f5f58d95070390092e0a3e314f62a7d101c2ba1d3d"} Nov 03 19:17:43.540507 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:43.540485 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" event={"ID":"604159f6-5047-43b8-b760-8083a8f2ef3c","Type":"ContainerStarted","Data":"f32b9c3e26d34e95e44ace8c1a43a30fcc463d04d5bd1b9d40f0ff8cb91a63fa"} Nov 03 19:17:43.673747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:43.673710 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:17:43.674026 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:43.673999 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:17:44.393725 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:44.393691 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:17:44.394079 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:44.393999 5019 patch_prober.go:29] interesting pod/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:17:44.394222 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:44.394201 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 19:17:44.549391 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:44.549359 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" event={"ID":"604159f6-5047-43b8-b760-8083a8f2ef3c","Type":"ContainerStarted","Data":"97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf"} Nov 03 19:17:44.549866 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:44.549837 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:17:44.679409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:44.674921 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:17:44.679409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:44.674978 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:17:44.696349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:44.691297 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" podStartSLOduration=2.773842136 podStartE2EDuration="3.691273558s" totalImagesPullingTime="917.431422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:17:44.677344851 +0000 UTC m=+1998.649906031" watchObservedRunningTime="2026-11-03 19:17:44.691273558 +0000 UTC m=+1998.663834741" Nov 03 19:17:45.396484 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:45.393852 5019 patch_prober.go:29] interesting pod/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:17:45.396484 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:45.393912 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 19:17:45.674978 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:45.674947 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:17:45.675611 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:45.675561 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:17:46.410350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:46.401803 5019 patch_prober.go:29] interesting pod/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:17:46.410350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:46.401883 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 19:17:46.683412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:46.677743 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:17:46.683412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:46.677815 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:17:47.413351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:47.405768 5019 patch_prober.go:29] interesting pod/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:17:47.413351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:47.405846 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 19:17:47.682362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:47.680403 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:17:47.682362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:47.680477 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:17:48.397354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:48.397190 5019 patch_prober.go:29] interesting pod/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:17:48.397354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:48.397261 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 19:17:48.697905 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:48.697873 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:17:48.698470 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:48.698440 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:17:48.699196 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:48.699177 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:17:49.398313 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:49.398280 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:17:49.398846 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:49.398718 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:17:49.399551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:49.399530 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:17:50.564685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:17:50.564644 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:18:05.485053 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.485013 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f"] Nov 03 19:18:05.490183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.490156 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.538871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.538842 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-826a68-openshift-default-dockercfg-72jsh\"" Nov 03 19:18:05.578012 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.577977 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-data\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.578215 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.578053 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-credential-socket\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.578215 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.578109 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-token\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.578215 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.578189 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-certs\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.578215 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.578212 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-envoy\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.578463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.578243 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istiod-ca-cert\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.578463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.578319 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-podinfo\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.578463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.578388 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-socket\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.578463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.578428 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h87z4\" (UniqueName: \"kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-kube-api-access-h87z4\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.591019 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.590985 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f"] Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.677634 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb"] Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.681830 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istiod-ca-cert\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.681944 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-podinfo\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.682007 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-socket\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.682064 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h87z4\" (UniqueName: \"kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-kube-api-access-h87z4\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.682157 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-data\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.682198 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-credential-socket\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.682657 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-token\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.682876 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-certs\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.682946 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-envoy\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.687917 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istiod-ca-cert\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.689829 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-credential-socket\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.690145 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-data\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.690432 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-socket\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.711894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.690951 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-certs\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.716697 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.716664 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-envoy\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.737401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.734240 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-podinfo\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.758345 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.758301 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:05.785355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.784304 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-maas-api-tls\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:05.785355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.784394 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ac426d-dc73-4cfa-8373-e7da9a598199-openshift-service-ca\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:05.785355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.784421 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-metrics-certs\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:05.785355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.784513 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnddr\" (UniqueName: \"kubernetes.io/projected/b8ac426d-dc73-4cfa-8373-e7da9a598199-kube-api-access-mnddr\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:05.810346 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.788714 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-token\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.810346 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.792556 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h87z4\" (UniqueName: \"kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-kube-api-access-h87z4\") pod \"e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.810346 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.808580 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:05.810346 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.809155 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1cc3f2-serving-cert\"" Nov 03 19:18:05.824049 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.823958 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb"] Nov 03 19:18:05.893938 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.893905 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-maas-api-tls\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:05.894126 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.893983 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ac426d-dc73-4cfa-8373-e7da9a598199-openshift-service-ca\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:05.894126 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.894015 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-metrics-certs\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:05.894245 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.894127 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnddr\" (UniqueName: \"kubernetes.io/projected/b8ac426d-dc73-4cfa-8373-e7da9a598199-kube-api-access-mnddr\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:05.894296 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:18:05.894226 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-serving-cert: secret "maas-api-e2e-api-b-1cc3f2-serving-cert" not found Nov 03 19:18:05.894436 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:18:05.894319 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-maas-api-tls podName:b8ac426d-dc73-4cfa-8373-e7da9a598199 nodeName:}" failed. No retries permitted until 2026-11-03 19:18:06.394292922 +0000 UTC m=+2020.366854104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-maas-api-tls") pod "maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" (UID: "b8ac426d-dc73-4cfa-8373-e7da9a598199") : secret "maas-api-e2e-api-b-1cc3f2-serving-cert" not found Nov 03 19:18:05.895211 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.895185 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ac426d-dc73-4cfa-8373-e7da9a598199-openshift-service-ca\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:05.897293 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:05.897272 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-metrics-certs\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:06.038434 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:06.036366 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnddr\" (UniqueName: \"kubernetes.io/projected/b8ac426d-dc73-4cfa-8373-e7da9a598199-kube-api-access-mnddr\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:06.430629 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:06.430594 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-maas-api-tls\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:06.433018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:06.432998 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-maas-api-tls\") pod \"maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:06.472289 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:06.472256 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f"] Nov 03 19:18:06.473947 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:18:06.473920 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c81f2ad_11c5_43a2_bde7_347f7e5f3715.slice/crio-e51f36a119ff7583cf3aeddeaafd92f70ce8eb5b0a13b8b5a803b52193562c59 WatchSource:0}: Error finding container e51f36a119ff7583cf3aeddeaafd92f70ce8eb5b0a13b8b5a803b52193562c59: Status 404 returned error can't find the container with id e51f36a119ff7583cf3aeddeaafd92f70ce8eb5b0a13b8b5a803b52193562c59 Nov 03 19:18:06.475961 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:06.475927 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:18:06.476046 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:06.475994 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:18:06.476046 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:06.476025 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:18:06.727022 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:06.726901 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:06.812835 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:06.812810 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" event={"ID":"6c81f2ad-11c5-43a2-bde7-347f7e5f3715","Type":"ContainerStarted","Data":"8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561"} Nov 03 19:18:06.812990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:06.812978 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" event={"ID":"6c81f2ad-11c5-43a2-bde7-347f7e5f3715","Type":"ContainerStarted","Data":"e51f36a119ff7583cf3aeddeaafd92f70ce8eb5b0a13b8b5a803b52193562c59"} Nov 03 19:18:06.928958 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:06.928910 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" podStartSLOduration=1.9288957839999998 podStartE2EDuration="1.928895784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:06.927440671 +0000 UTC m=+2020.900001852" watchObservedRunningTime="2026-11-03 19:18:06.928895784 +0000 UTC m=+2020.901456966" Nov 03 19:18:07.146738 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:07.145267 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb"] Nov 03 19:18:07.811382 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:07.811280 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:07.831096 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:07.830762 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:07.836877 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:07.836836 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" event={"ID":"b8ac426d-dc73-4cfa-8373-e7da9a598199","Type":"ContainerStarted","Data":"4564758dc8268d837f516e03021b07fc440ee560053abab8e601aa22fcfbfc54"} Nov 03 19:18:07.837639 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:07.837074 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:07.838361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:07.838312 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:18:08.846018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:08.845984 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" event={"ID":"b8ac426d-dc73-4cfa-8373-e7da9a598199","Type":"ContainerStarted","Data":"016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4"} Nov 03 19:18:08.846554 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:08.846207 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:08.959666 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:08.959614 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" podStartSLOduration=2.959933959 podStartE2EDuration="3.959599354s" totalImagesPullingTime="999.665395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:08.955305745 +0000 UTC m=+2022.927866928" watchObservedRunningTime="2026-11-03 19:18:08.959599354 +0000 UTC m=+2022.932160535" Nov 03 19:18:11.301895 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:18:11.301855 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:18:15.017967 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:15.017930 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:18:28.186528 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:28.186496 5019 generic.go:356] "Generic (PLEG): container finished" podID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerID="6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989" exitCode=0 Nov 03 19:18:28.186988 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:28.186569 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" event={"ID":"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da","Type":"ContainerDied","Data":"6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989"} Nov 03 19:18:28.187034 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:28.186992 5019 scope.go:122] "RemoveContainer" containerID="6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989" Nov 03 19:18:28.188861 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:28.188826 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:18:28.189000 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:28.188898 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:18:28.189000 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:28.188929 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:18:29.259924 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:29.259880 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" event={"ID":"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da","Type":"ContainerStarted","Data":"7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862"} Nov 03 19:18:29.634274 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:29.620194 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:18:29.644834 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:29.644798 5019 patch_prober.go:29] interesting pod/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 19:18:29.645467 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:29.645429 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 19:18:30.619732 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:30.619700 5019 patch_prober.go:29] interesting pod/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 19:18:30.620361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:30.620314 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 19:18:31.423959 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:31.423912 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:18:31.618909 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:31.618870 5019 patch_prober.go:29] interesting pod/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 19:18:31.619115 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:31.618929 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 19:18:32.618798 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:32.618763 5019 patch_prober.go:29] interesting pod/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 19:18:32.619434 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:32.618819 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 19:18:33.619346 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:33.618753 5019 patch_prober.go:29] interesting pod/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 19:18:33.619346 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:33.618820 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 19:18:34.618687 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:34.618652 5019 patch_prober.go:29] interesting pod/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 19:18:34.618957 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:34.618928 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 19:18:35.623923 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:35.623888 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:18:35.625000 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:18:35.624982 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:18:41.299357 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:18:41.299288 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:19:11.366254 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:19:11.366209 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:19:14.836692 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:14.836649 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf"] Nov 03 19:19:14.860354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:14.842511 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" podUID="dffda2a7-8f0f-42a5-b6df-5b656f40ee22" containerName="maas-api" containerID="cri-o://bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881" gracePeriod=30 Nov 03 19:19:14.983266 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:14.983222 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" podUID="dffda2a7-8f0f-42a5-b6df-5b656f40ee22" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.116:8443/health\": dial tcp 10.132.0.116:8443: connect: connection refused" Nov 03 19:19:15.527017 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.526985 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:19:15.610798 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.610705 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-maas-api-tls\") pod \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " Nov 03 19:19:15.610798 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.610771 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-openshift-service-ca\") pod \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " Nov 03 19:19:15.611036 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.610821 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-kube-api-access-n25hh\" (UniqueName: \"kubernetes.io/projected/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-kube-api-access-n25hh\") pod \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " Nov 03 19:19:15.611036 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.610855 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-metrics-certs\") pod \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\" (UID: \"dffda2a7-8f0f-42a5-b6df-5b656f40ee22\") " Nov 03 19:19:15.611295 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.611265 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-openshift-service-ca" pod "dffda2a7-8f0f-42a5-b6df-5b656f40ee22" (UID: "dffda2a7-8f0f-42a5-b6df-5b656f40ee22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:15.611700 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.611673 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:15.613463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.613433 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-maas-api-tls" pod "dffda2a7-8f0f-42a5-b6df-5b656f40ee22" (UID: "dffda2a7-8f0f-42a5-b6df-5b656f40ee22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:15.614485 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.614451 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-metrics-certs" pod "dffda2a7-8f0f-42a5-b6df-5b656f40ee22" (UID: "dffda2a7-8f0f-42a5-b6df-5b656f40ee22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:15.614622 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.614604 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-kube-api-access-n25hh" pod "dffda2a7-8f0f-42a5-b6df-5b656f40ee22" (UID: "dffda2a7-8f0f-42a5-b6df-5b656f40ee22"). InnerVolumeSpecName "kube-api-access-n25hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:15.671542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.671506 5019 generic.go:356] "Generic (PLEG): container finished" podID="dffda2a7-8f0f-42a5-b6df-5b656f40ee22" containerID="bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881" exitCode=0 Nov 03 19:19:15.671726 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.671566 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" event={"ID":"dffda2a7-8f0f-42a5-b6df-5b656f40ee22","Type":"ContainerDied","Data":"bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881"} Nov 03 19:19:15.671726 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.671599 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" Nov 03 19:19:15.671726 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.671614 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf" event={"ID":"dffda2a7-8f0f-42a5-b6df-5b656f40ee22","Type":"ContainerDied","Data":"fcde4d550e4eb7a9bc406ccc7a84fd6902f62e1a73175aadac52a430b2bdc15f"} Nov 03 19:19:15.671726 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.671638 5019 scope.go:122] "RemoveContainer" containerID="bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881" Nov 03 19:19:15.695536 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.685477 5019 scope.go:122] "RemoveContainer" containerID="bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881" Nov 03 19:19:15.695536 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:19:15.685773 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881\": container with ID starting with bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881 not found: ID does not exist" containerID="bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881" Nov 03 19:19:15.695536 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.685807 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881"} err="failed to get container status \"bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881\": rpc error: code = NotFound desc = could not find container \"bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881\": container with ID starting with bef579ede01d04592090aad7bedbc7f2844d351b1b337418ffdd32c1f319f881 not found: ID does not exist" Nov 03 19:19:15.713407 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.713377 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n25hh\" (UniqueName: \"kubernetes.io/projected/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-kube-api-access-n25hh\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:15.713560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.713411 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:15.713560 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.713435 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffda2a7-8f0f-42a5-b6df-5b656f40ee22-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:15.873401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.872759 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf"] Nov 03 19:19:15.892417 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:15.892381 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4702dd-7b4796d544-zr4gf"] Nov 03 19:19:16.152303 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:16.152225 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dffda2a7-8f0f-42a5-b6df-5b656f40ee22" path="/var/lib/kubelet/pods/dffda2a7-8f0f-42a5-b6df-5b656f40ee22/volumes" Nov 03 19:19:24.545088 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:24.545047 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg"] Nov 03 19:19:24.545658 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:24.545268 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" containerID="cri-o://c49a2befcdc89260f9c66d7f08768c6764fd11e29b05af0498e9940be276b775" gracePeriod=30 Nov 03 19:19:28.621084 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:28.621041 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:19:28.621536 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:28.621108 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:19:29.442803 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:29.442770 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk"] Nov 03 19:19:29.543599 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:29.543564 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw"] Nov 03 19:19:29.719838 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:29.719758 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:19:29.720318 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:29.719985 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:19:29.831459 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:29.831420 5019 generic.go:356] "Generic (PLEG): container finished" podID="b8a00172-f381-4936-8800-18bdd9f428d6" containerID="c49a2befcdc89260f9c66d7f08768c6764fd11e29b05af0498e9940be276b775" exitCode=0 Nov 03 19:19:29.831640 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:29.831480 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" event={"ID":"b8a00172-f381-4936-8800-18bdd9f428d6","Type":"ContainerDied","Data":"c49a2befcdc89260f9c66d7f08768c6764fd11e29b05af0498e9940be276b775"} Nov 03 19:19:29.831640 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:29.831535 5019 scope.go:122] "RemoveContainer" containerID="6dbdeb05a98c74bcf6a1bf259446849ac996f006ca2d585294941a6e88a57709" Nov 03 19:19:30.061307 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.060943 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:19:30.148259 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148228 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-socket\") pod \"b8a00172-f381-4936-8800-18bdd9f428d6\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " Nov 03 19:19:30.148464 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148271 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-kube-api-access-f927r\" (UniqueName: \"kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-kube-api-access-f927r\") pod \"b8a00172-f381-4936-8800-18bdd9f428d6\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " Nov 03 19:19:30.148464 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148313 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-credential-socket\") pod \"b8a00172-f381-4936-8800-18bdd9f428d6\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " Nov 03 19:19:30.148464 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148387 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a00172-f381-4936-8800-18bdd9f428d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a00172-f381-4936-8800-18bdd9f428d6-istiod-ca-cert\") pod \"b8a00172-f381-4936-8800-18bdd9f428d6\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " Nov 03 19:19:30.148464 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148422 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-istio-token\" (UniqueName: \"kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-istio-token\") pod \"b8a00172-f381-4936-8800-18bdd9f428d6\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " Nov 03 19:19:30.148464 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148451 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-envoy\") pod \"b8a00172-f381-4936-8800-18bdd9f428d6\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " Nov 03 19:19:30.148751 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148480 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-data\") pod \"b8a00172-f381-4936-8800-18bdd9f428d6\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " Nov 03 19:19:30.148751 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148548 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-socket" pod "b8a00172-f381-4936-8800-18bdd9f428d6" (UID: "b8a00172-f381-4936-8800-18bdd9f428d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:30.148751 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148672 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8a00172-f381-4936-8800-18bdd9f428d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a00172-f381-4936-8800-18bdd9f428d6-istio-podinfo\") pod \"b8a00172-f381-4936-8800-18bdd9f428d6\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " Nov 03 19:19:30.148751 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148743 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-certs\") pod \"b8a00172-f381-4936-8800-18bdd9f428d6\" (UID: \"b8a00172-f381-4936-8800-18bdd9f428d6\") " Nov 03 19:19:30.148975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148783 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-credential-socket" pod "b8a00172-f381-4936-8800-18bdd9f428d6" (UID: "b8a00172-f381-4936-8800-18bdd9f428d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:30.148975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148873 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a00172-f381-4936-8800-18bdd9f428d6-istiod-ca-cert" pod "b8a00172-f381-4936-8800-18bdd9f428d6" (UID: "b8a00172-f381-4936-8800-18bdd9f428d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:30.149081 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.148972 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-data" pod "b8a00172-f381-4936-8800-18bdd9f428d6" (UID: "b8a00172-f381-4936-8800-18bdd9f428d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:30.149176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.149151 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-certs" pod "b8a00172-f381-4936-8800-18bdd9f428d6" (UID: "b8a00172-f381-4936-8800-18bdd9f428d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:30.149467 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.149423 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:30.149467 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.149449 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:30.149467 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.149464 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:30.149681 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.149479 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:30.149681 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.149493 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a00172-f381-4936-8800-18bdd9f428d6-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:30.150801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.150765 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-envoy" pod "b8a00172-f381-4936-8800-18bdd9f428d6" (UID: "b8a00172-f381-4936-8800-18bdd9f428d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:30.150975 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.150918 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-kube-api-access-f927r" pod "b8a00172-f381-4936-8800-18bdd9f428d6" (UID: "b8a00172-f381-4936-8800-18bdd9f428d6"). InnerVolumeSpecName "kube-api-access-f927r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:30.151098 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.151012 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-istio-token" pod "b8a00172-f381-4936-8800-18bdd9f428d6" (UID: "b8a00172-f381-4936-8800-18bdd9f428d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:30.151268 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.151253 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8a00172-f381-4936-8800-18bdd9f428d6-istio-podinfo" pod "b8a00172-f381-4936-8800-18bdd9f428d6" (UID: "b8a00172-f381-4936-8800-18bdd9f428d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:19:30.250779 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.250682 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f927r\" (UniqueName: \"kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-kube-api-access-f927r\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:30.250779 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.250721 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a00172-f381-4936-8800-18bdd9f428d6-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:30.250779 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.250739 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a00172-f381-4936-8800-18bdd9f428d6-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:30.250779 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.250753 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a00172-f381-4936-8800-18bdd9f428d6-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:30.884350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.879125 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" event={"ID":"b8a00172-f381-4936-8800-18bdd9f428d6","Type":"ContainerDied","Data":"2e91302933abba8e8101be331b798d1578bbdfffacbed60142f21dda151a0af2"} Nov 03 19:19:30.884350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.879196 5019 scope.go:122] "RemoveContainer" containerID="c49a2befcdc89260f9c66d7f08768c6764fd11e29b05af0498e9940be276b775" Nov 03 19:19:30.884350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:30.879342 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg" Nov 03 19:19:31.102387 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:31.098858 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg"] Nov 03 19:19:31.170416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:31.170384 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4702dd-openshift-default-9cf5ddd6f-xwvsg"] Nov 03 19:19:32.152835 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:32.152804 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" path="/var/lib/kubelet/pods/b8a00172-f381-4936-8800-18bdd9f428d6/volumes" Nov 03 19:19:34.443118 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.443079 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" podUID="1ededdd5-4be7-413c-8fb9-6be120d8119a" containerName="payload-processing" containerID="cri-o://9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c" gracePeriod=25 Nov 03 19:19:34.544832 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.544787 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" podUID="201aa634-3a94-4fb1-ab67-8f11817e6a7a" containerName="payload-pre-processing" containerID="cri-o://5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f" gracePeriod=25 Nov 03 19:19:34.780618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.780588 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" Nov 03 19:19:34.859449 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.859423 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" Nov 03 19:19:34.859683 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.859659 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ededdd5-4be7-413c-8fb9-6be120d8119a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ededdd5-4be7-413c-8fb9-6be120d8119a-plugins-config-volume\") pod \"1ededdd5-4be7-413c-8fb9-6be120d8119a\" (UID: \"1ededdd5-4be7-413c-8fb9-6be120d8119a\") " Nov 03 19:19:34.859879 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.859861 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ededdd5-4be7-413c-8fb9-6be120d8119a-kube-api-access-2ffpd\" (UniqueName: \"kubernetes.io/projected/1ededdd5-4be7-413c-8fb9-6be120d8119a-kube-api-access-2ffpd\") pod \"1ededdd5-4be7-413c-8fb9-6be120d8119a\" (UID: \"1ededdd5-4be7-413c-8fb9-6be120d8119a\") " Nov 03 19:19:34.860061 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.860038 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ededdd5-4be7-413c-8fb9-6be120d8119a-plugins-config-volume" pod "1ededdd5-4be7-413c-8fb9-6be120d8119a" (UID: "1ededdd5-4be7-413c-8fb9-6be120d8119a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:34.860890 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.860777 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ededdd5-4be7-413c-8fb9-6be120d8119a-plugins-config-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:34.862653 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.862622 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ededdd5-4be7-413c-8fb9-6be120d8119a-kube-api-access-2ffpd" pod "1ededdd5-4be7-413c-8fb9-6be120d8119a" (UID: "1ededdd5-4be7-413c-8fb9-6be120d8119a"). InnerVolumeSpecName "kube-api-access-2ffpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:34.937603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.937565 5019 generic.go:356] "Generic (PLEG): container finished" podID="201aa634-3a94-4fb1-ab67-8f11817e6a7a" containerID="5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f" exitCode=0 Nov 03 19:19:34.937777 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.937613 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" Nov 03 19:19:34.937777 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.937645 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" event={"ID":"201aa634-3a94-4fb1-ab67-8f11817e6a7a","Type":"ContainerDied","Data":"5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f"} Nov 03 19:19:34.937777 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.937702 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw" event={"ID":"201aa634-3a94-4fb1-ab67-8f11817e6a7a","Type":"ContainerDied","Data":"d000dc1996f9a3a41aadbccedb768117d90900dd0449946759f64479f7186227"} Nov 03 19:19:34.937777 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.937731 5019 scope.go:122] "RemoveContainer" containerID="5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f" Nov 03 19:19:34.939594 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.939566 5019 generic.go:356] "Generic (PLEG): container finished" podID="1ededdd5-4be7-413c-8fb9-6be120d8119a" containerID="9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c" exitCode=0 Nov 03 19:19:34.939705 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.939658 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" event={"ID":"1ededdd5-4be7-413c-8fb9-6be120d8119a","Type":"ContainerDied","Data":"9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c"} Nov 03 19:19:34.939705 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.939697 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" event={"ID":"1ededdd5-4be7-413c-8fb9-6be120d8119a","Type":"ContainerDied","Data":"9f9cf64f86a0387e6d98a417ac0a018551cfecae1defe33f22aea4dfc90304ce"} Nov 03 19:19:34.939796 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.939762 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk" Nov 03 19:19:34.951144 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.951109 5019 scope.go:122] "RemoveContainer" containerID="5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f" Nov 03 19:19:34.951451 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:19:34.951433 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f\": container with ID starting with 5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f not found: ID does not exist" containerID="5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f" Nov 03 19:19:34.951547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.951459 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f"} err="failed to get container status \"5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f\": rpc error: code = NotFound desc = could not find container \"5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f\": container with ID starting with 5d75e0e46342da545c0a1ea6a4f1ab35955199a5608940a739a4d111a866684f not found: ID does not exist" Nov 03 19:19:34.951547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.951480 5019 scope.go:122] "RemoveContainer" containerID="9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c" Nov 03 19:19:34.962294 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.962271 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201aa634-3a94-4fb1-ab67-8f11817e6a7a-kube-api-access-rhlgf\" (UniqueName: \"kubernetes.io/projected/201aa634-3a94-4fb1-ab67-8f11817e6a7a-kube-api-access-rhlgf\") pod \"201aa634-3a94-4fb1-ab67-8f11817e6a7a\" (UID: \"201aa634-3a94-4fb1-ab67-8f11817e6a7a\") " Nov 03 19:19:34.962584 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.962436 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/201aa634-3a94-4fb1-ab67-8f11817e6a7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201aa634-3a94-4fb1-ab67-8f11817e6a7a-plugins-config-volume\") pod \"201aa634-3a94-4fb1-ab67-8f11817e6a7a\" (UID: \"201aa634-3a94-4fb1-ab67-8f11817e6a7a\") " Nov 03 19:19:34.963028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.963008 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ffpd\" (UniqueName: \"kubernetes.io/projected/1ededdd5-4be7-413c-8fb9-6be120d8119a-kube-api-access-2ffpd\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:34.963608 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.963580 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201aa634-3a94-4fb1-ab67-8f11817e6a7a-plugins-config-volume" pod "201aa634-3a94-4fb1-ab67-8f11817e6a7a" (UID: "201aa634-3a94-4fb1-ab67-8f11817e6a7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:34.963938 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.963917 5019 scope.go:122] "RemoveContainer" containerID="9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c" Nov 03 19:19:34.964386 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:19:34.964248 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c\": container with ID starting with 9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c not found: ID does not exist" containerID="9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c" Nov 03 19:19:34.964386 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.964279 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c"} err="failed to get container status \"9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c\": rpc error: code = NotFound desc = could not find container \"9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c\": container with ID starting with 9e1bb0b0fef1d986e466bba8d6aa9e8fe0a5d8b9a6a0fec6409c8501e7fa6f5c not found: ID does not exist" Nov 03 19:19:34.965745 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:34.965376 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201aa634-3a94-4fb1-ab67-8f11817e6a7a-kube-api-access-rhlgf" pod "201aa634-3a94-4fb1-ab67-8f11817e6a7a" (UID: "201aa634-3a94-4fb1-ab67-8f11817e6a7a"). InnerVolumeSpecName "kube-api-access-rhlgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:35.065009 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:35.064914 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhlgf\" (UniqueName: \"kubernetes.io/projected/201aa634-3a94-4fb1-ab67-8f11817e6a7a-kube-api-access-rhlgf\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:35.065009 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:35.064954 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201aa634-3a94-4fb1-ab67-8f11817e6a7a-plugins-config-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:35.108037 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:35.108001 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk"] Nov 03 19:19:35.112592 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:35.112568 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d4656-76d544f864-rgwhk"] Nov 03 19:19:35.354817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:35.354734 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw"] Nov 03 19:19:35.358420 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:35.358394 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d4656-7cd957bd7f-qsmzw"] Nov 03 19:19:35.847625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:35.847584 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp"] Nov 03 19:19:35.948629 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:35.948591 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g"] Nov 03 19:19:36.310367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:36.308078 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ededdd5-4be7-413c-8fb9-6be120d8119a" path="/var/lib/kubelet/pods/1ededdd5-4be7-413c-8fb9-6be120d8119a/volumes" Nov 03 19:19:36.323062 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:36.323004 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201aa634-3a94-4fb1-ab67-8f11817e6a7a" path="/var/lib/kubelet/pods/201aa634-3a94-4fb1-ab67-8f11817e6a7a/volumes" Nov 03 19:19:40.848734 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:40.848678 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" podUID="d6f63cce-6768-472d-bdc7-89b0ef2abd56" containerName="payload-processing" containerID="cri-o://76b7aa35b56c7c8e043fb11e0da5a73767d53cdfd29e844675dd489a995214f3" gracePeriod=25 Nov 03 19:19:40.949500 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:40.949439 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" podUID="5258c74f-5b40-44b8-8693-00ccaf309394" containerName="payload-pre-processing" containerID="cri-o://27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820" gracePeriod=25 Nov 03 19:19:41.026535 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.026503 5019 generic.go:356] "Generic (PLEG): container finished" podID="d6f63cce-6768-472d-bdc7-89b0ef2abd56" containerID="76b7aa35b56c7c8e043fb11e0da5a73767d53cdfd29e844675dd489a995214f3" exitCode=0 Nov 03 19:19:41.026693 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.026578 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" event={"ID":"d6f63cce-6768-472d-bdc7-89b0ef2abd56","Type":"ContainerDied","Data":"76b7aa35b56c7c8e043fb11e0da5a73767d53cdfd29e844675dd489a995214f3"} Nov 03 19:19:41.294032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.293646 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" Nov 03 19:19:41.327844 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:19:41.327810 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:19:41.417072 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.417048 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" Nov 03 19:19:41.426172 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.426141 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f63cce-6768-472d-bdc7-89b0ef2abd56-kube-api-access-sprxx\" (UniqueName: \"kubernetes.io/projected/d6f63cce-6768-472d-bdc7-89b0ef2abd56-kube-api-access-sprxx\") pod \"d6f63cce-6768-472d-bdc7-89b0ef2abd56\" (UID: \"d6f63cce-6768-472d-bdc7-89b0ef2abd56\") " Nov 03 19:19:41.426381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.426321 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f63cce-6768-472d-bdc7-89b0ef2abd56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f63cce-6768-472d-bdc7-89b0ef2abd56-plugins-config-volume\") pod \"d6f63cce-6768-472d-bdc7-89b0ef2abd56\" (UID: \"d6f63cce-6768-472d-bdc7-89b0ef2abd56\") " Nov 03 19:19:41.426895 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.426850 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f63cce-6768-472d-bdc7-89b0ef2abd56-plugins-config-volume" pod "d6f63cce-6768-472d-bdc7-89b0ef2abd56" (UID: "d6f63cce-6768-472d-bdc7-89b0ef2abd56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:41.427171 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.427133 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f63cce-6768-472d-bdc7-89b0ef2abd56-plugins-config-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:41.429283 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.429256 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f63cce-6768-472d-bdc7-89b0ef2abd56-kube-api-access-sprxx" pod "d6f63cce-6768-472d-bdc7-89b0ef2abd56" (UID: "d6f63cce-6768-472d-bdc7-89b0ef2abd56"). InnerVolumeSpecName "kube-api-access-sprxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:41.527958 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.527877 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5258c74f-5b40-44b8-8693-00ccaf309394-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5258c74f-5b40-44b8-8693-00ccaf309394-plugins-config-volume\") pod \"5258c74f-5b40-44b8-8693-00ccaf309394\" (UID: \"5258c74f-5b40-44b8-8693-00ccaf309394\") " Nov 03 19:19:41.528182 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.528166 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5258c74f-5b40-44b8-8693-00ccaf309394-kube-api-access-c2rlj\" (UniqueName: \"kubernetes.io/projected/5258c74f-5b40-44b8-8693-00ccaf309394-kube-api-access-c2rlj\") pod \"5258c74f-5b40-44b8-8693-00ccaf309394\" (UID: \"5258c74f-5b40-44b8-8693-00ccaf309394\") " Nov 03 19:19:41.528382 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.528322 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5258c74f-5b40-44b8-8693-00ccaf309394-plugins-config-volume" pod "5258c74f-5b40-44b8-8693-00ccaf309394" (UID: "5258c74f-5b40-44b8-8693-00ccaf309394"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:41.528770 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.528741 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sprxx\" (UniqueName: \"kubernetes.io/projected/d6f63cce-6768-472d-bdc7-89b0ef2abd56-kube-api-access-sprxx\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:41.528770 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.528771 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5258c74f-5b40-44b8-8693-00ccaf309394-plugins-config-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:41.530816 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.530790 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5258c74f-5b40-44b8-8693-00ccaf309394-kube-api-access-c2rlj" pod "5258c74f-5b40-44b8-8693-00ccaf309394" (UID: "5258c74f-5b40-44b8-8693-00ccaf309394"). InnerVolumeSpecName "kube-api-access-c2rlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:41.630315 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:41.630282 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2rlj\" (UniqueName: \"kubernetes.io/projected/5258c74f-5b40-44b8-8693-00ccaf309394-kube-api-access-c2rlj\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:42.036087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.036061 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" Nov 03 19:19:42.036551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.036101 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp" event={"ID":"d6f63cce-6768-472d-bdc7-89b0ef2abd56","Type":"ContainerDied","Data":"f62925eec488ac82be693bf086aeeba184c143ee9fd2568d4a07c07d5a0206a0"} Nov 03 19:19:42.036551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.036168 5019 scope.go:122] "RemoveContainer" containerID="76b7aa35b56c7c8e043fb11e0da5a73767d53cdfd29e844675dd489a995214f3" Nov 03 19:19:42.037799 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.037762 5019 generic.go:356] "Generic (PLEG): container finished" podID="5258c74f-5b40-44b8-8693-00ccaf309394" containerID="27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820" exitCode=0 Nov 03 19:19:42.037912 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.037812 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" Nov 03 19:19:42.037912 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.037809 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" event={"ID":"5258c74f-5b40-44b8-8693-00ccaf309394","Type":"ContainerDied","Data":"27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820"} Nov 03 19:19:42.038034 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.037948 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g" event={"ID":"5258c74f-5b40-44b8-8693-00ccaf309394","Type":"ContainerDied","Data":"edbc928ccd47003f635853e64c768a6fb807587925caf19e8db79ea819e491f4"} Nov 03 19:19:42.048722 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.048707 5019 scope.go:122] "RemoveContainer" containerID="27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820" Nov 03 19:19:42.058462 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.058440 5019 scope.go:122] "RemoveContainer" containerID="27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820" Nov 03 19:19:42.058738 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:19:42.058719 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820\": container with ID starting with 27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820 not found: ID does not exist" containerID="27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820" Nov 03 19:19:42.058779 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.058748 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820"} err="failed to get container status \"27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820\": rpc error: code = NotFound desc = could not find container \"27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820\": container with ID starting with 27ab9ebc16fc85ad234f6cc43f104e1a9b6dcd31c73da6c57b5bfc13a8053820 not found: ID does not exist" Nov 03 19:19:42.196102 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.196061 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp"] Nov 03 19:19:42.202235 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.202204 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4f1a90-5999899db8-wtjvp"] Nov 03 19:19:42.308814 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.308783 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g"] Nov 03 19:19:42.343352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:42.343305 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f1a90-5c98dfbf5f-x4g5g"] Nov 03 19:19:44.154633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:44.154585 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5258c74f-5b40-44b8-8693-00ccaf309394" path="/var/lib/kubelet/pods/5258c74f-5b40-44b8-8693-00ccaf309394/volumes" Nov 03 19:19:44.155220 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:44.155199 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f63cce-6768-472d-bdc7-89b0ef2abd56" path="/var/lib/kubelet/pods/d6f63cce-6768-472d-bdc7-89b0ef2abd56/volumes" Nov 03 19:19:46.078186 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:46.078156 5019 generic.go:356] "Generic (PLEG): container finished" podID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerID="e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c" exitCode=0 Nov 03 19:19:46.078745 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:46.078383 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" event={"ID":"96db2630-ed5b-49f8-b716-53ffd18dfa33","Type":"ContainerDied","Data":"e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c"} Nov 03 19:19:46.078818 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:46.078802 5019 scope.go:122] "RemoveContainer" containerID="e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c" Nov 03 19:19:46.080910 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:46.080871 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:19:46.081015 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:46.080964 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:19:46.081015 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:46.081008 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:19:46.389737 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:46.389632 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x"] Nov 03 19:19:46.390274 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:46.389929 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" podUID="5c0a76ec-c204-473d-8b44-1bb35920dceb" containerName="maas-api" containerID="cri-o://7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6" gracePeriod=30 Nov 03 19:19:46.977379 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:46.963925 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:19:47.098303 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.098268 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" event={"ID":"96db2630-ed5b-49f8-b716-53ffd18dfa33","Type":"ContainerStarted","Data":"903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957"} Nov 03 19:19:47.100870 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.100842 5019 generic.go:356] "Generic (PLEG): container finished" podID="5c0a76ec-c204-473d-8b44-1bb35920dceb" containerID="7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6" exitCode=0 Nov 03 19:19:47.101024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.100883 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" Nov 03 19:19:47.101024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.100893 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" event={"ID":"5c0a76ec-c204-473d-8b44-1bb35920dceb","Type":"ContainerDied","Data":"7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6"} Nov 03 19:19:47.101024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.100933 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x" event={"ID":"5c0a76ec-c204-473d-8b44-1bb35920dceb","Type":"ContainerDied","Data":"92a89a5bd6046f0cb88647ccd6c003ca42fecfdc2ff56528cad023a25cd91aad"} Nov 03 19:19:47.101024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.100962 5019 scope.go:122] "RemoveContainer" containerID="7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6" Nov 03 19:19:47.118309 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.118173 5019 scope.go:122] "RemoveContainer" containerID="7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6" Nov 03 19:19:47.118797 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:19:47.118726 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6\": container with ID starting with 7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6 not found: ID does not exist" containerID="7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6" Nov 03 19:19:47.118797 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.118768 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6"} err="failed to get container status \"7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6\": rpc error: code = NotFound desc = could not find container \"7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6\": container with ID starting with 7c029c9b6ca8e2afadf1ba3a2383fc880a47dc1f1cc9ea4598a9a91b522bb6f6 not found: ID does not exist" Nov 03 19:19:47.193350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.173713 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0a76ec-c204-473d-8b44-1bb35920dceb-kube-api-access-dph8t\" (UniqueName: \"kubernetes.io/projected/5c0a76ec-c204-473d-8b44-1bb35920dceb-kube-api-access-dph8t\") pod \"5c0a76ec-c204-473d-8b44-1bb35920dceb\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " Nov 03 19:19:47.193350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.173779 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-metrics-certs\") pod \"5c0a76ec-c204-473d-8b44-1bb35920dceb\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " Nov 03 19:19:47.193350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.175578 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-maas-api-tls\") pod \"5c0a76ec-c204-473d-8b44-1bb35920dceb\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " Nov 03 19:19:47.193350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.175672 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0a76ec-c204-473d-8b44-1bb35920dceb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0a76ec-c204-473d-8b44-1bb35920dceb-openshift-service-ca\") pod \"5c0a76ec-c204-473d-8b44-1bb35920dceb\" (UID: \"5c0a76ec-c204-473d-8b44-1bb35920dceb\") " Nov 03 19:19:47.193350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.177971 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0a76ec-c204-473d-8b44-1bb35920dceb-openshift-service-ca" pod "5c0a76ec-c204-473d-8b44-1bb35920dceb" (UID: "5c0a76ec-c204-473d-8b44-1bb35920dceb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:47.193350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.180605 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-maas-api-tls" pod "5c0a76ec-c204-473d-8b44-1bb35920dceb" (UID: "5c0a76ec-c204-473d-8b44-1bb35920dceb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:47.193350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.180641 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0a76ec-c204-473d-8b44-1bb35920dceb-kube-api-access-dph8t" pod "5c0a76ec-c204-473d-8b44-1bb35920dceb" (UID: "5c0a76ec-c204-473d-8b44-1bb35920dceb"). InnerVolumeSpecName "kube-api-access-dph8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:47.227380 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.222484 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-metrics-certs" pod "5c0a76ec-c204-473d-8b44-1bb35920dceb" (UID: "5c0a76ec-c204-473d-8b44-1bb35920dceb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:47.284350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.282157 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:47.284350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.282198 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0a76ec-c204-473d-8b44-1bb35920dceb-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:47.284350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.282214 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dph8t\" (UniqueName: \"kubernetes.io/projected/5c0a76ec-c204-473d-8b44-1bb35920dceb-kube-api-access-dph8t\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:47.284350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.282230 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0a76ec-c204-473d-8b44-1bb35920dceb-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:47.383351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.374022 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:19:47.383351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.374152 5019 patch_prober.go:29] interesting pod/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:19:47.383351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.374198 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 19:19:47.568639 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.568506 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x"] Nov 03 19:19:47.680223 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:47.642927 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f1a90-79cccfdc8c-drh8x"] Nov 03 19:19:48.153292 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.153251 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c0a76ec-c204-473d-8b44-1bb35920dceb" path="/var/lib/kubelet/pods/5c0a76ec-c204-473d-8b44-1bb35920dceb/volumes" Nov 03 19:19:48.382352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.381705 5019 patch_prober.go:29] interesting pod/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:19:48.382352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.381773 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 19:19:48.457187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.457160 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9"] Nov 03 19:19:48.457903 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.457887 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5258c74f-5b40-44b8-8693-00ccaf309394" containerName="payload-pre-processing" Nov 03 19:19:48.457950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.457904 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5258c74f-5b40-44b8-8693-00ccaf309394" containerName="payload-pre-processing" Nov 03 19:19:48.457950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.457924 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" Nov 03 19:19:48.457950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.457930 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" Nov 03 19:19:48.457950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.457942 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5c0a76ec-c204-473d-8b44-1bb35920dceb" containerName="maas-api" Nov 03 19:19:48.457950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.457946 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0a76ec-c204-473d-8b44-1bb35920dceb" containerName="maas-api" Nov 03 19:19:48.458114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.457964 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d6f63cce-6768-472d-bdc7-89b0ef2abd56" containerName="payload-processing" Nov 03 19:19:48.458114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.457971 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f63cce-6768-472d-bdc7-89b0ef2abd56" containerName="payload-processing" Nov 03 19:19:48.458114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.457995 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1ededdd5-4be7-413c-8fb9-6be120d8119a" containerName="payload-processing" Nov 03 19:19:48.458114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458003 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ededdd5-4be7-413c-8fb9-6be120d8119a" containerName="payload-processing" Nov 03 19:19:48.458114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458029 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dffda2a7-8f0f-42a5-b6df-5b656f40ee22" containerName="maas-api" Nov 03 19:19:48.458114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458037 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffda2a7-8f0f-42a5-b6df-5b656f40ee22" containerName="maas-api" Nov 03 19:19:48.458114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458051 5019 cpu_manager.go:414] "containerMap: removing container" podUID="201aa634-3a94-4fb1-ab67-8f11817e6a7a" containerName="payload-pre-processing" Nov 03 19:19:48.458114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458056 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201aa634-3a94-4fb1-ab67-8f11817e6a7a" containerName="payload-pre-processing" Nov 03 19:19:48.458399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458162 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0a76ec-c204-473d-8b44-1bb35920dceb" containerName="maas-api" Nov 03 19:19:48.458399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458174 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f63cce-6768-472d-bdc7-89b0ef2abd56" containerName="payload-processing" Nov 03 19:19:48.458399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458181 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ededdd5-4be7-413c-8fb9-6be120d8119a" containerName="payload-processing" Nov 03 19:19:48.458399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458186 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" Nov 03 19:19:48.458399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458194 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffda2a7-8f0f-42a5-b6df-5b656f40ee22" containerName="maas-api" Nov 03 19:19:48.458399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458200 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="201aa634-3a94-4fb1-ab67-8f11817e6a7a" containerName="payload-pre-processing" Nov 03 19:19:48.458399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458208 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5258c74f-5b40-44b8-8693-00ccaf309394" containerName="payload-pre-processing" Nov 03 19:19:48.458399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.458214 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" Nov 03 19:19:48.474357 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.473593 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:48.520103 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.520071 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9"] Nov 03 19:19:48.540747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.540716 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1410ed-serving-cert\"" Nov 03 19:19:48.613302 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.613263 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f448ccd6-0521-480f-8c09-04d31e3858f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:48.613519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.613355 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wst6j\" (UniqueName: \"kubernetes.io/projected/f448ccd6-0521-480f-8c09-04d31e3858f2-kube-api-access-wst6j\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:48.613519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.613442 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-metrics-certs\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:48.613646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.613555 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:48.716396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.714696 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:48.716396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.714900 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f448ccd6-0521-480f-8c09-04d31e3858f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:48.716396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.714957 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wst6j\" (UniqueName: \"kubernetes.io/projected/f448ccd6-0521-480f-8c09-04d31e3858f2-kube-api-access-wst6j\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:48.716396 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.715043 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-metrics-certs\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:48.724407 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.718721 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f448ccd6-0521-480f-8c09-04d31e3858f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:48.724407 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:19:48.718840 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-serving-cert: secret "maas-api-e2e-inf-a-1410ed-serving-cert" not found Nov 03 19:19:48.724407 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:19:48.718901 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-maas-api-tls podName:f448ccd6-0521-480f-8c09-04d31e3858f2 nodeName:}" failed. No retries permitted until 2026-11-03 19:19:49.218881612 +0000 UTC m=+2123.191442797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-maas-api-tls") pod "maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" (UID: "f448ccd6-0521-480f-8c09-04d31e3858f2") : secret "maas-api-e2e-inf-a-1410ed-serving-cert" not found Nov 03 19:19:48.732965 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.732924 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-metrics-certs\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:48.841375 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:48.840927 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wst6j\" (UniqueName: \"kubernetes.io/projected/f448ccd6-0521-480f-8c09-04d31e3858f2-kube-api-access-wst6j\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:49.286351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:49.285614 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:49.298845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:49.298797 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:49.364353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:49.363630 5019 patch_prober.go:29] interesting pod/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:19:49.364353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:49.363699 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 19:19:49.401226 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:49.401185 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:49.799231 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:49.798973 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9"] Nov 03 19:19:49.802195 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:19:49.802136 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf448ccd6_0521_480f_8c09_04d31e3858f2.slice/crio-32f7a670c5bc3626ed5329d36d6d001e28775fb7d91bf80290675268c04beca7 WatchSource:0}: Error finding container 32f7a670c5bc3626ed5329d36d6d001e28775fb7d91bf80290675268c04beca7: Status 404 returned error can't find the container with id 32f7a670c5bc3626ed5329d36d6d001e28775fb7d91bf80290675268c04beca7 Nov 03 19:19:50.153529 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:50.153450 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" event={"ID":"f448ccd6-0521-480f-8c09-04d31e3858f2","Type":"ContainerStarted","Data":"32f7a670c5bc3626ed5329d36d6d001e28775fb7d91bf80290675268c04beca7"} Nov 03 19:19:50.365743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:50.365702 5019 patch_prober.go:29] interesting pod/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:19:50.366296 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:50.366265 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 19:19:51.344448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:51.344410 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" event={"ID":"f448ccd6-0521-480f-8c09-04d31e3858f2","Type":"ContainerStarted","Data":"63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a"} Nov 03 19:19:51.344728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:51.344710 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:19:51.382354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:51.379912 5019 patch_prober.go:29] interesting pod/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:19:51.382354 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:51.379988 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 19:19:51.456112 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:51.456066 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" podStartSLOduration=2.429117405 podStartE2EDuration="3.456052076s" totalImagesPullingTime="1.026934671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:19:51.455473477 +0000 UTC m=+2125.428034660" watchObservedRunningTime="2026-11-03 19:19:51.456052076 +0000 UTC m=+2125.428613266" Nov 03 19:19:52.375940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:52.375897 5019 patch_prober.go:29] interesting pod/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:19:52.376247 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:52.376203 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 19:19:53.370541 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:53.370494 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:19:53.371273 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:53.371242 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:19:53.372308 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:53.372285 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:19:56.080910 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:56.080876 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl"] Nov 03 19:19:56.081383 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:56.081099 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" containerID="cri-o://01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3" gracePeriod=30 Nov 03 19:19:57.398388 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:19:57.398320 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:20:01.382251 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.382218 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:20:01.422277 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422235 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-kube-api-access-jk94r\" (UniqueName: \"kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-kube-api-access-jk94r\") pod \"ad088e1c-645f-4365-89ac-6e8f85589ddb\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " Nov 03 19:20:01.422277 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422278 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-data\") pod \"ad088e1c-645f-4365-89ac-6e8f85589ddb\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " Nov 03 19:20:01.422581 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422361 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-envoy\") pod \"ad088e1c-645f-4365-89ac-6e8f85589ddb\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " Nov 03 19:20:01.422581 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422534 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-certs\") pod \"ad088e1c-645f-4365-89ac-6e8f85589ddb\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " Nov 03 19:20:01.422702 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422606 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-credential-socket\") pod \"ad088e1c-645f-4365-89ac-6e8f85589ddb\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " Nov 03 19:20:01.422702 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422654 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-data" pod "ad088e1c-645f-4365-89ac-6e8f85589ddb" (UID: "ad088e1c-645f-4365-89ac-6e8f85589ddb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:01.422702 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422662 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-socket\") pod \"ad088e1c-645f-4365-89ac-6e8f85589ddb\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " Nov 03 19:20:01.422868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422769 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad088e1c-645f-4365-89ac-6e8f85589ddb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad088e1c-645f-4365-89ac-6e8f85589ddb-istiod-ca-cert\") pod \"ad088e1c-645f-4365-89ac-6e8f85589ddb\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " Nov 03 19:20:01.422868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422804 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-token\" (UniqueName: \"kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-token\") pod \"ad088e1c-645f-4365-89ac-6e8f85589ddb\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " Nov 03 19:20:01.422868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422795 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-certs" pod "ad088e1c-645f-4365-89ac-6e8f85589ddb" (UID: "ad088e1c-645f-4365-89ac-6e8f85589ddb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:01.422868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422852 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-podinfo\") pod \"ad088e1c-645f-4365-89ac-6e8f85589ddb\" (UID: \"ad088e1c-645f-4365-89ac-6e8f85589ddb\") " Nov 03 19:20:01.423082 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422898 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-socket" pod "ad088e1c-645f-4365-89ac-6e8f85589ddb" (UID: "ad088e1c-645f-4365-89ac-6e8f85589ddb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:01.423204 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.423180 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad088e1c-645f-4365-89ac-6e8f85589ddb-istiod-ca-cert" pod "ad088e1c-645f-4365-89ac-6e8f85589ddb" (UID: "ad088e1c-645f-4365-89ac-6e8f85589ddb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:01.423262 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.422856 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-credential-socket" pod "ad088e1c-645f-4365-89ac-6e8f85589ddb" (UID: "ad088e1c-645f-4365-89ac-6e8f85589ddb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:01.423661 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.423638 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:01.423776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.423665 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:01.423776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.423682 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:01.423776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.423697 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:01.423776 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.423710 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad088e1c-645f-4365-89ac-6e8f85589ddb-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:01.424936 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.424906 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-kube-api-access-jk94r" pod "ad088e1c-645f-4365-89ac-6e8f85589ddb" (UID: "ad088e1c-645f-4365-89ac-6e8f85589ddb"). InnerVolumeSpecName "kube-api-access-jk94r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:01.425022 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.424974 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-podinfo" pod "ad088e1c-645f-4365-89ac-6e8f85589ddb" (UID: "ad088e1c-645f-4365-89ac-6e8f85589ddb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:20:01.425076 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.425050 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-envoy" pod "ad088e1c-645f-4365-89ac-6e8f85589ddb" (UID: "ad088e1c-645f-4365-89ac-6e8f85589ddb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:01.425240 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.425219 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-token" pod "ad088e1c-645f-4365-89ac-6e8f85589ddb" (UID: "ad088e1c-645f-4365-89ac-6e8f85589ddb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:01.502151 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.502120 5019 generic.go:356] "Generic (PLEG): container finished" podID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerID="01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3" exitCode=0 Nov 03 19:20:01.502314 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.502177 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" Nov 03 19:20:01.502314 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.502271 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" event={"ID":"ad088e1c-645f-4365-89ac-6e8f85589ddb","Type":"ContainerDied","Data":"01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3"} Nov 03 19:20:01.502502 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.502461 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl" event={"ID":"ad088e1c-645f-4365-89ac-6e8f85589ddb","Type":"ContainerDied","Data":"785cfcdc5c6bc68e4111118bfe6cf9da6809e4ec26eaf2984f8cbeafbe5726c3"} Nov 03 19:20:01.502548 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.502535 5019 scope.go:122] "RemoveContainer" containerID="01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3" Nov 03 19:20:01.514548 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.514530 5019 scope.go:122] "RemoveContainer" containerID="3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de" Nov 03 19:20:01.524669 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.524649 5019 scope.go:122] "RemoveContainer" containerID="01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3" Nov 03 19:20:01.524918 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:20:01.524900 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3\": container with ID starting with 01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3 not found: ID does not exist" containerID="01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3" Nov 03 19:20:01.524990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.524926 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3"} err="failed to get container status \"01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3\": rpc error: code = NotFound desc = could not find container \"01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3\": container with ID starting with 01634cf9167c46468d42730a39514a6cf5f9c6784df5a44ee69f8de0a0b405a3 not found: ID does not exist" Nov 03 19:20:01.524990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.524943 5019 scope.go:122] "RemoveContainer" containerID="3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de" Nov 03 19:20:01.525248 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:20:01.525171 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de\": container with ID starting with 3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de not found: ID does not exist" containerID="3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de" Nov 03 19:20:01.525248 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.525206 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de"} err="failed to get container status \"3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de\": rpc error: code = NotFound desc = could not find container \"3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de\": container with ID starting with 3cb86ccba661a3a24fcc08b6374a729591445fbd6b92e84808659fc5c95962de not found: ID does not exist" Nov 03 19:20:01.525553 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.525536 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:01.525649 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.525556 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:01.525649 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.525567 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk94r\" (UniqueName: \"kubernetes.io/projected/ad088e1c-645f-4365-89ac-6e8f85589ddb-kube-api-access-jk94r\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:01.525649 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.525578 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad088e1c-645f-4365-89ac-6e8f85589ddb-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:01.650442 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.650300 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl"] Nov 03 19:20:01.655248 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:01.655221 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f1a90-openshift-default-7c58d44ccf-cg5hl"] Nov 03 19:20:02.153465 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:02.153432 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" path="/var/lib/kubelet/pods/ad088e1c-645f-4365-89ac-6e8f85589ddb/volumes" Nov 03 19:20:08.481520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:08.481482 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6"] Nov 03 19:20:08.482081 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:08.481731 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" podUID="1d55ab25-8642-4b74-99c1-9aaf2eed7a15" containerName="maas-api" containerID="cri-o://eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5" gracePeriod=30 Nov 03 19:20:09.074539 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.074513 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:20:09.266102 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.266069 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-kube-api-access-g2rbm\" (UniqueName: \"kubernetes.io/projected/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-kube-api-access-g2rbm\") pod \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " Nov 03 19:20:09.266294 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.266122 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-openshift-service-ca\") pod \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " Nov 03 19:20:09.266294 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.266244 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-maas-api-tls\") pod \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " Nov 03 19:20:09.266520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.266492 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-metrics-certs\") pod \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\" (UID: \"1d55ab25-8642-4b74-99c1-9aaf2eed7a15\") " Nov 03 19:20:09.266601 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.266523 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-openshift-service-ca" pod "1d55ab25-8642-4b74-99c1-9aaf2eed7a15" (UID: "1d55ab25-8642-4b74-99c1-9aaf2eed7a15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:09.267855 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.267827 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:09.268976 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.268954 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-maas-api-tls" pod "1d55ab25-8642-4b74-99c1-9aaf2eed7a15" (UID: "1d55ab25-8642-4b74-99c1-9aaf2eed7a15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:20:09.269068 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.269012 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-kube-api-access-g2rbm" pod "1d55ab25-8642-4b74-99c1-9aaf2eed7a15" (UID: "1d55ab25-8642-4b74-99c1-9aaf2eed7a15"). InnerVolumeSpecName "kube-api-access-g2rbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:09.269191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.269173 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-metrics-certs" pod "1d55ab25-8642-4b74-99c1-9aaf2eed7a15" (UID: "1d55ab25-8642-4b74-99c1-9aaf2eed7a15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:20:09.370128 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.369318 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:09.370128 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.369384 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:09.370128 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.369402 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2rbm\" (UniqueName: \"kubernetes.io/projected/1d55ab25-8642-4b74-99c1-9aaf2eed7a15-kube-api-access-g2rbm\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:09.632102 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.632021 5019 generic.go:356] "Generic (PLEG): container finished" podID="1d55ab25-8642-4b74-99c1-9aaf2eed7a15" containerID="eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5" exitCode=0 Nov 03 19:20:09.632102 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.632065 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" Nov 03 19:20:09.632603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.632102 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" event={"ID":"1d55ab25-8642-4b74-99c1-9aaf2eed7a15","Type":"ContainerDied","Data":"eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5"} Nov 03 19:20:09.632603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.632158 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6" event={"ID":"1d55ab25-8642-4b74-99c1-9aaf2eed7a15","Type":"ContainerDied","Data":"221bac659712ee5811e418cc52ac42ab6f541d3d7f8e52dc76ca08048ebe3fb7"} Nov 03 19:20:09.632603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.632180 5019 scope.go:122] "RemoveContainer" containerID="eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5" Nov 03 19:20:09.644498 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.644478 5019 scope.go:122] "RemoveContainer" containerID="eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5" Nov 03 19:20:09.644820 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:20:09.644798 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5\": container with ID starting with eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5 not found: ID does not exist" containerID="eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5" Nov 03 19:20:09.644864 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.644832 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5"} err="failed to get container status \"eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5\": rpc error: code = NotFound desc = could not find container \"eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5\": container with ID starting with eaea5e5f3289e8f0021bc585d028538f578a0b9eecb142e103ec768eead71df5 not found: ID does not exist" Nov 03 19:20:09.790586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.790552 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6"] Nov 03 19:20:09.834745 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:09.834715 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-ttml6"] Nov 03 19:20:10.153481 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:10.153450 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d55ab25-8642-4b74-99c1-9aaf2eed7a15" path="/var/lib/kubelet/pods/1d55ab25-8642-4b74-99c1-9aaf2eed7a15/volumes" Nov 03 19:20:11.300314 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:20:11.300278 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:20:29.793275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:29.793246 5019 generic.go:356] "Generic (PLEG): container finished" podID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerID="8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561" exitCode=0 Nov 03 19:20:29.793719 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:29.793355 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" event={"ID":"6c81f2ad-11c5-43a2-bde7-347f7e5f3715","Type":"ContainerDied","Data":"8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561"} Nov 03 19:20:29.793849 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:29.793827 5019 scope.go:122] "RemoveContainer" containerID="8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561" Nov 03 19:20:29.795637 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:29.795609 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:20:29.795722 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:29.795681 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:20:29.795722 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:29.795715 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:20:30.814115 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:30.814074 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" event={"ID":"6c81f2ad-11c5-43a2-bde7-347f7e5f3715","Type":"ContainerStarted","Data":"2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720"} Nov 03 19:20:31.809585 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:31.809541 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:20:31.809810 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:31.809776 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:20:31.809876 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:31.809837 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 19:20:32.810056 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:32.810021 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:20:32.810605 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:32.810084 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 19:20:33.809683 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:33.809646 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:20:33.809871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:33.809703 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 19:20:34.809863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:34.809819 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:20:34.810383 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:34.809888 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 19:20:35.809991 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:35.809948 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:20:35.810432 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:35.810020 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 19:20:36.814264 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:36.814236 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:20:36.814777 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:36.814750 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:20:36.815419 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:36.815398 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:20:38.977245 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:38.977214 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5"] Nov 03 19:20:38.977764 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:38.977470 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" podUID="604159f6-5047-43b8-b760-8083a8f2ef3c" containerName="maas-api" containerID="cri-o://97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf" gracePeriod=30 Nov 03 19:20:39.459243 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.459210 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:20:39.548428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.548395 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-maas-api-tls\") pod \"604159f6-5047-43b8-b760-8083a8f2ef3c\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " Nov 03 19:20:39.548617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.548578 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604159f6-5047-43b8-b760-8083a8f2ef3c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604159f6-5047-43b8-b760-8083a8f2ef3c-openshift-service-ca\") pod \"604159f6-5047-43b8-b760-8083a8f2ef3c\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " Nov 03 19:20:39.548689 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.548651 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604159f6-5047-43b8-b760-8083a8f2ef3c-kube-api-access-654qn\" (UniqueName: \"kubernetes.io/projected/604159f6-5047-43b8-b760-8083a8f2ef3c-kube-api-access-654qn\") pod \"604159f6-5047-43b8-b760-8083a8f2ef3c\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " Nov 03 19:20:39.548804 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.548780 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-metrics-certs\") pod \"604159f6-5047-43b8-b760-8083a8f2ef3c\" (UID: \"604159f6-5047-43b8-b760-8083a8f2ef3c\") " Nov 03 19:20:39.549022 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.548995 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604159f6-5047-43b8-b760-8083a8f2ef3c-openshift-service-ca" pod "604159f6-5047-43b8-b760-8083a8f2ef3c" (UID: "604159f6-5047-43b8-b760-8083a8f2ef3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:39.549454 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.549432 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604159f6-5047-43b8-b760-8083a8f2ef3c-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:39.550888 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.550863 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-maas-api-tls" pod "604159f6-5047-43b8-b760-8083a8f2ef3c" (UID: "604159f6-5047-43b8-b760-8083a8f2ef3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:20:39.550999 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.550900 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604159f6-5047-43b8-b760-8083a8f2ef3c-kube-api-access-654qn" pod "604159f6-5047-43b8-b760-8083a8f2ef3c" (UID: "604159f6-5047-43b8-b760-8083a8f2ef3c"). InnerVolumeSpecName "kube-api-access-654qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:39.551183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.551164 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-metrics-certs" pod "604159f6-5047-43b8-b760-8083a8f2ef3c" (UID: "604159f6-5047-43b8-b760-8083a8f2ef3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:20:39.652641 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.652592 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654qn\" (UniqueName: \"kubernetes.io/projected/604159f6-5047-43b8-b760-8083a8f2ef3c-kube-api-access-654qn\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:39.652885 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.652857 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:39.652987 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.652976 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604159f6-5047-43b8-b760-8083a8f2ef3c-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:39.880478 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.880390 5019 generic.go:356] "Generic (PLEG): container finished" podID="604159f6-5047-43b8-b760-8083a8f2ef3c" containerID="97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf" exitCode=0 Nov 03 19:20:39.880478 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.880441 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" Nov 03 19:20:39.880721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.880472 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" event={"ID":"604159f6-5047-43b8-b760-8083a8f2ef3c","Type":"ContainerDied","Data":"97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf"} Nov 03 19:20:39.880721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.880561 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5" event={"ID":"604159f6-5047-43b8-b760-8083a8f2ef3c","Type":"ContainerDied","Data":"f32b9c3e26d34e95e44ace8c1a43a30fcc463d04d5bd1b9d40f0ff8cb91a63fa"} Nov 03 19:20:39.880721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.880592 5019 scope.go:122] "RemoveContainer" containerID="97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf" Nov 03 19:20:39.898030 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.898011 5019 scope.go:122] "RemoveContainer" containerID="97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf" Nov 03 19:20:39.898643 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:20:39.898618 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf\": container with ID starting with 97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf not found: ID does not exist" containerID="97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf" Nov 03 19:20:39.898923 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:39.898895 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf"} err="failed to get container status \"97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf\": rpc error: code = NotFound desc = could not find container \"97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf\": container with ID starting with 97f4857600211faaccaf2be8b49c1125681e19ee8f8682408d28db0c7c2e47bf not found: ID does not exist" Nov 03 19:20:40.091747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:40.091677 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5"] Nov 03 19:20:40.100069 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:40.097792 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4656-84d9d8c595-6wdw5"] Nov 03 19:20:40.159042 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:40.158952 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604159f6-5047-43b8-b760-8083a8f2ef3c" path="/var/lib/kubelet/pods/604159f6-5047-43b8-b760-8083a8f2ef3c/volumes" Nov 03 19:20:41.298788 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:20:41.298756 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:20:48.198672 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:48.198638 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582"] Nov 03 19:20:48.199461 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:48.199421 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" containerID="cri-o://7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862" gracePeriod=30 Nov 03 19:20:50.626471 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:50.626429 5019 patch_prober.go:29] interesting pod/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:20:50.626966 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:50.626507 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:20:51.115004 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:51.114967 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m"] Nov 03 19:20:51.115190 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:51.115169 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerName="istio-proxy" containerID="cri-o://caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c" gracePeriod=30 Nov 03 19:20:53.486160 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.486135 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:20:53.589159 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.589130 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istiod-ca-cert\") pod \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " Nov 03 19:20:53.589376 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.589177 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-socket\") pod \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " Nov 03 19:20:53.589376 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.589212 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-credential-socket\") pod \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " Nov 03 19:20:53.589376 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.589254 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-podinfo\") pod \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " Nov 03 19:20:53.589376 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.589317 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-kube-api-access-j7knv\" (UniqueName: \"kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-kube-api-access-j7knv\") pod \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " Nov 03 19:20:53.589625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.589395 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-envoy\") pod \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " Nov 03 19:20:53.589625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.589443 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-certs\") pod \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " Nov 03 19:20:53.589625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.589492 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-data\") pod \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " Nov 03 19:20:53.589625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.589575 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istiod-ca-cert" pod "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" (UID: "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:53.589625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.589582 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-socket" pod "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" (UID: "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:53.589863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.589656 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-token\" (UniqueName: \"kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-token\") pod \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\" (UID: \"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da\") " Nov 03 19:20:53.590071 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.590030 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-data" pod "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" (UID: "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:53.590225 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.590098 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-credential-socket" pod "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" (UID: "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:53.590225 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.590138 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-certs" pod "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" (UID: "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:53.590459 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.590438 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:53.590551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.590464 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:53.590551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.590479 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:53.590551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.590493 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:53.590551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.590507 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:53.592510 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.592478 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-kube-api-access-j7knv" pod "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" (UID: "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da"). InnerVolumeSpecName "kube-api-access-j7knv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:53.592510 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.592501 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-token" pod "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" (UID: "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:53.592649 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.592588 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-envoy" pod "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" (UID: "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:53.593764 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.593738 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-podinfo" pod "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" (UID: "7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:20:53.692686 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.692594 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:53.692686 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.692632 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7knv\" (UniqueName: \"kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-kube-api-access-j7knv\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:53.692686 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.692648 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:53.692686 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:53.692661 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:54.031969 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:54.031938 5019 generic.go:356] "Generic (PLEG): container finished" podID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerID="7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862" exitCode=0 Nov 03 19:20:54.032146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:54.031999 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" Nov 03 19:20:54.032146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:54.032030 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" event={"ID":"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da","Type":"ContainerDied","Data":"7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862"} Nov 03 19:20:54.032146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:54.032103 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582" event={"ID":"7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da","Type":"ContainerDied","Data":"1bd83a061e34e11346c90cadff7e5c389ecea8e9526b32d02abe791a6ce0293e"} Nov 03 19:20:54.032146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:54.032128 5019 scope.go:122] "RemoveContainer" containerID="7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862" Nov 03 19:20:54.043102 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:54.043078 5019 scope.go:122] "RemoveContainer" containerID="6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989" Nov 03 19:20:54.053686 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:54.053668 5019 scope.go:122] "RemoveContainer" containerID="7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862" Nov 03 19:20:54.053968 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:20:54.053949 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862\": container with ID starting with 7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862 not found: ID does not exist" containerID="7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862" Nov 03 19:20:54.054049 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:54.053981 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862"} err="failed to get container status \"7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862\": rpc error: code = NotFound desc = could not find container \"7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862\": container with ID starting with 7d2803ecbac69f5be94d2a9bf4d58253836787648f8c8b7e6bf70d394e236862 not found: ID does not exist" Nov 03 19:20:54.054049 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:54.054005 5019 scope.go:122] "RemoveContainer" containerID="6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989" Nov 03 19:20:54.054256 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:20:54.054239 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989\": container with ID starting with 6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989 not found: ID does not exist" containerID="6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989" Nov 03 19:20:54.054311 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:54.054263 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989"} err="failed to get container status \"6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989\": rpc error: code = NotFound desc = could not find container \"6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989\": container with ID starting with 6b5e316768fd222bb3efd5157f1876fdf51291720d72d55923b60c63c7c60989 not found: ID does not exist" Nov 03 19:20:54.190540 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:54.190510 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582"] Nov 03 19:20:54.196165 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:54.196140 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4656-openshift-default-5f476bb87f-hw582"] Nov 03 19:20:56.153888 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.153857 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" path="/var/lib/kubelet/pods/7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da/volumes" Nov 03 19:20:56.428690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.428644 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:20:56.537147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.537115 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-kube-api-access-rngjm\" (UniqueName: \"kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-kube-api-access-rngjm\") pod \"6d590414-a23c-4b37-840b-47f06e7b83b5\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " Nov 03 19:20:56.537388 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.537313 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-socket\") pod \"6d590414-a23c-4b37-840b-47f06e7b83b5\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " Nov 03 19:20:56.537542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.537525 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-podinfo\") pod \"6d590414-a23c-4b37-840b-47f06e7b83b5\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " Nov 03 19:20:56.537625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.537592 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-credential-socket\") pod \"6d590414-a23c-4b37-840b-47f06e7b83b5\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " Nov 03 19:20:56.537680 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.537646 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-token\" (UniqueName: \"kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-token\") pod \"6d590414-a23c-4b37-840b-47f06e7b83b5\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " Nov 03 19:20:56.537680 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.537659 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-socket" pod "6d590414-a23c-4b37-840b-47f06e7b83b5" (UID: "6d590414-a23c-4b37-840b-47f06e7b83b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.537785 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.537685 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d590414-a23c-4b37-840b-47f06e7b83b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d590414-a23c-4b37-840b-47f06e7b83b5-istiod-ca-cert\") pod \"6d590414-a23c-4b37-840b-47f06e7b83b5\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " Nov 03 19:20:56.537785 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.537719 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-envoy\") pod \"6d590414-a23c-4b37-840b-47f06e7b83b5\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " Nov 03 19:20:56.537882 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.537810 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-certs\") pod \"6d590414-a23c-4b37-840b-47f06e7b83b5\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " Nov 03 19:20:56.537936 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.537870 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-credential-socket" pod "6d590414-a23c-4b37-840b-47f06e7b83b5" (UID: "6d590414-a23c-4b37-840b-47f06e7b83b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.537936 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.537883 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-data\") pod \"6d590414-a23c-4b37-840b-47f06e7b83b5\" (UID: \"6d590414-a23c-4b37-840b-47f06e7b83b5\") " Nov 03 19:20:56.538388 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.538252 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d590414-a23c-4b37-840b-47f06e7b83b5-istiod-ca-cert" pod "6d590414-a23c-4b37-840b-47f06e7b83b5" (UID: "6d590414-a23c-4b37-840b-47f06e7b83b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:56.538886 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.538756 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-data" pod "6d590414-a23c-4b37-840b-47f06e7b83b5" (UID: "6d590414-a23c-4b37-840b-47f06e7b83b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.538886 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.538806 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-certs" pod "6d590414-a23c-4b37-840b-47f06e7b83b5" (UID: "6d590414-a23c-4b37-840b-47f06e7b83b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.539203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.539184 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.539292 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.539205 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.539292 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.539214 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d590414-a23c-4b37-840b-47f06e7b83b5-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.539292 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.539227 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.539292 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.539238 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.540321 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.540294 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-kube-api-access-rngjm" pod "6d590414-a23c-4b37-840b-47f06e7b83b5" (UID: "6d590414-a23c-4b37-840b-47f06e7b83b5"). InnerVolumeSpecName "kube-api-access-rngjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:56.541244 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.541219 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-envoy" pod "6d590414-a23c-4b37-840b-47f06e7b83b5" (UID: "6d590414-a23c-4b37-840b-47f06e7b83b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:56.541454 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.541430 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-podinfo" pod "6d590414-a23c-4b37-840b-47f06e7b83b5" (UID: "6d590414-a23c-4b37-840b-47f06e7b83b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:20:56.542062 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.542043 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-token" pod "6d590414-a23c-4b37-840b-47f06e7b83b5" (UID: "6d590414-a23c-4b37-840b-47f06e7b83b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:56.640365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.640310 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.640365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.640365 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.640588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.640375 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d590414-a23c-4b37-840b-47f06e7b83b5-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:56.640588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:56.640385 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rngjm\" (UniqueName: \"kubernetes.io/projected/6d590414-a23c-4b37-840b-47f06e7b83b5-kube-api-access-rngjm\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:57.057950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:57.057912 5019 generic.go:356] "Generic (PLEG): container finished" podID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerID="caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c" exitCode=0 Nov 03 19:20:57.058140 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:57.057963 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" event={"ID":"6d590414-a23c-4b37-840b-47f06e7b83b5","Type":"ContainerDied","Data":"caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c"} Nov 03 19:20:57.058140 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:57.057972 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" Nov 03 19:20:57.058140 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:57.058022 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m" event={"ID":"6d590414-a23c-4b37-840b-47f06e7b83b5","Type":"ContainerDied","Data":"39912ea8b6078a26d0d8511855bc1965c3b881e95142e95f4d9de19831e85371"} Nov 03 19:20:57.058140 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:57.058053 5019 scope.go:122] "RemoveContainer" containerID="caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c" Nov 03 19:20:57.072965 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:57.072947 5019 scope.go:122] "RemoveContainer" containerID="caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c" Nov 03 19:20:57.076512 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:20:57.076485 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c\": container with ID starting with caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c not found: ID does not exist" containerID="caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c" Nov 03 19:20:57.076615 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:57.076523 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c"} err="failed to get container status \"caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c\": rpc error: code = NotFound desc = could not find container \"caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c\": container with ID starting with caeb448550b00c2a88bcf70d63671d2cb6c1ab1c2ebe481db11a83a411e4519c not found: ID does not exist" Nov 03 19:20:57.138759 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:20:57.138718 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d590414_a23c_4b37_840b_47f06e7b83b5.slice/crio-39912ea8b6078a26d0d8511855bc1965c3b881e95142e95f4d9de19831e85371\": RecentStats: unable to find data in memory cache]" Nov 03 19:20:57.219564 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:57.219529 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m"] Nov 03 19:20:57.224142 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:57.224113 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xj77m"] Nov 03 19:20:58.151219 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:20:58.151184 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" path="/var/lib/kubelet/pods/6d590414-a23c-4b37-840b-47f06e7b83b5/volumes" Nov 03 19:21:11.299335 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:21:11.299225 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:21:16.690136 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:16.690096 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm"] Nov 03 19:21:16.690798 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:16.690740 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" podUID="313d392e-831e-428c-ad0d-0bbd03674212" containerName="maas-api" containerID="cri-o://311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413" gracePeriod=30 Nov 03 19:21:16.989523 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:16.989496 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:21:17.146477 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.146440 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313d392e-831e-428c-ad0d-0bbd03674212-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/313d392e-831e-428c-ad0d-0bbd03674212-openshift-service-ca\") pod \"313d392e-831e-428c-ad0d-0bbd03674212\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " Nov 03 19:21:17.146717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.146549 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-maas-api-tls\") pod \"313d392e-831e-428c-ad0d-0bbd03674212\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " Nov 03 19:21:17.146717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.146667 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313d392e-831e-428c-ad0d-0bbd03674212-kube-api-access-grvfd\" (UniqueName: \"kubernetes.io/projected/313d392e-831e-428c-ad0d-0bbd03674212-kube-api-access-grvfd\") pod \"313d392e-831e-428c-ad0d-0bbd03674212\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " Nov 03 19:21:17.146864 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.146815 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-metrics-certs\" (UniqueName: \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-metrics-certs\") pod \"313d392e-831e-428c-ad0d-0bbd03674212\" (UID: \"313d392e-831e-428c-ad0d-0bbd03674212\") " Nov 03 19:21:17.146953 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.146863 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313d392e-831e-428c-ad0d-0bbd03674212-openshift-service-ca" pod "313d392e-831e-428c-ad0d-0bbd03674212" (UID: "313d392e-831e-428c-ad0d-0bbd03674212"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:21:17.147465 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.147439 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/313d392e-831e-428c-ad0d-0bbd03674212-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:17.151518 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.151458 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-metrics-certs" pod "313d392e-831e-428c-ad0d-0bbd03674212" (UID: "313d392e-831e-428c-ad0d-0bbd03674212"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:21:17.151518 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.151482 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-maas-api-tls" pod "313d392e-831e-428c-ad0d-0bbd03674212" (UID: "313d392e-831e-428c-ad0d-0bbd03674212"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:21:17.151518 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.151504 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313d392e-831e-428c-ad0d-0bbd03674212-kube-api-access-grvfd" pod "313d392e-831e-428c-ad0d-0bbd03674212" (UID: "313d392e-831e-428c-ad0d-0bbd03674212"). InnerVolumeSpecName "kube-api-access-grvfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:17.251449 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.249556 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:17.251449 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.249591 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/313d392e-831e-428c-ad0d-0bbd03674212-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:17.251449 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.249610 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grvfd\" (UniqueName: \"kubernetes.io/projected/313d392e-831e-428c-ad0d-0bbd03674212-kube-api-access-grvfd\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:17.271431 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.271387 5019 generic.go:356] "Generic (PLEG): container finished" podID="313d392e-831e-428c-ad0d-0bbd03674212" containerID="311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413" exitCode=0 Nov 03 19:21:17.271618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.271466 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" Nov 03 19:21:17.271618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.271479 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" event={"ID":"313d392e-831e-428c-ad0d-0bbd03674212","Type":"ContainerDied","Data":"311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413"} Nov 03 19:21:17.271618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.271590 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm" event={"ID":"313d392e-831e-428c-ad0d-0bbd03674212","Type":"ContainerDied","Data":"6f7b3b53484d2caa4edbaa5911b59031a44359c39ef4ad3f6fb3fe17d29ef832"} Nov 03 19:21:17.271788 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.271622 5019 scope.go:122] "RemoveContainer" containerID="311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413" Nov 03 19:21:17.286091 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.286070 5019 scope.go:122] "RemoveContainer" containerID="311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413" Nov 03 19:21:17.286428 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:21:17.286398 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413\": container with ID starting with 311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413 not found: ID does not exist" containerID="311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413" Nov 03 19:21:17.286518 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.286439 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413"} err="failed to get container status \"311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413\": rpc error: code = NotFound desc = could not find container \"311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413\": container with ID starting with 311dec669de4c615787018074d03ad8511bc776cc40ec8052a92ca41b5cb8413 not found: ID does not exist" Nov 03 19:21:17.465605 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.465567 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm"] Nov 03 19:21:17.469164 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:17.469130 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-144890-7ccc9f48b5-6lcmm"] Nov 03 19:21:18.151883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.151854 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313d392e-831e-428c-ad0d-0bbd03674212" path="/var/lib/kubelet/pods/313d392e-831e-428c-ad0d-0bbd03674212/volumes" Nov 03 19:21:18.428134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428104 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9"] Nov 03 19:21:18.428861 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428840 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" Nov 03 19:21:18.428861 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428858 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428868 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428873 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428884 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerName="istio-proxy" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428889 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerName="istio-proxy" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428895 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1d55ab25-8642-4b74-99c1-9aaf2eed7a15" containerName="maas-api" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428899 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d55ab25-8642-4b74-99c1-9aaf2eed7a15" containerName="maas-api" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428909 5019 cpu_manager.go:414] "containerMap: removing container" podUID="604159f6-5047-43b8-b760-8083a8f2ef3c" containerName="maas-api" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428915 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604159f6-5047-43b8-b760-8083a8f2ef3c" containerName="maas-api" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428943 5019 cpu_manager.go:414] "containerMap: removing container" podUID="313d392e-831e-428c-ad0d-0bbd03674212" containerName="maas-api" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428948 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313d392e-831e-428c-ad0d-0bbd03674212" containerName="maas-api" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428956 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428960 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428968 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428972 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428988 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" Nov 03 19:21:18.429011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.428992 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a00172-f381-4936-8800-18bdd9f428d6" containerName="istio-proxy" Nov 03 19:21:18.429542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.429085 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="313d392e-831e-428c-ad0d-0bbd03674212" containerName="maas-api" Nov 03 19:21:18.429542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.429094 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" Nov 03 19:21:18.429542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.429099 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7cc1ef-c4de-4e81-9bbe-ad1c30db88da" containerName="istio-proxy" Nov 03 19:21:18.429542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.429110 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" Nov 03 19:21:18.429542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.429117 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad088e1c-645f-4365-89ac-6e8f85589ddb" containerName="istio-proxy" Nov 03 19:21:18.429542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.429122 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d590414-a23c-4b37-840b-47f06e7b83b5" containerName="istio-proxy" Nov 03 19:21:18.429542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.429130 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d55ab25-8642-4b74-99c1-9aaf2eed7a15" containerName="maas-api" Nov 03 19:21:18.429542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.429136 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="604159f6-5047-43b8-b760-8083a8f2ef3c" containerName="maas-api" Nov 03 19:21:18.434251 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.434226 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:18.530412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.530378 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d2cbe3-serving-cert\"" Nov 03 19:21:18.534133 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.534107 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9"] Nov 03 19:21:18.588349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.587809 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-maas-api-tls\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:18.588349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.588044 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-metrics-certs\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:18.588349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.588080 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8r49\" (UniqueName: \"kubernetes.io/projected/e494384c-192a-4c1e-8bd8-5c0389817cc5-kube-api-access-r8r49\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:18.595399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.591504 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e494384c-192a-4c1e-8bd8-5c0389817cc5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:18.730095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.728494 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-maas-api-tls\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:18.730095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.728822 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-metrics-certs\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:18.730095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.728956 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8r49\" (UniqueName: \"kubernetes.io/projected/e494384c-192a-4c1e-8bd8-5c0389817cc5-kube-api-access-r8r49\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:18.730095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.729041 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e494384c-192a-4c1e-8bd8-5c0389817cc5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:18.730095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.730005 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e494384c-192a-4c1e-8bd8-5c0389817cc5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:18.730624 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:21:18.730120 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-serving-cert: secret "maas-api-e2e-api-a-d2cbe3-serving-cert" not found Nov 03 19:21:18.730624 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:21:18.730191 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-maas-api-tls podName:e494384c-192a-4c1e-8bd8-5c0389817cc5 nodeName:}" failed. No retries permitted until 2026-11-03 19:21:19.230171576 +0000 UTC m=+2213.202732757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-maas-api-tls") pod "maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" (UID: "e494384c-192a-4c1e-8bd8-5c0389817cc5") : secret "maas-api-e2e-api-a-d2cbe3-serving-cert" not found Nov 03 19:21:18.741932 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.741746 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-metrics-certs\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:18.791696 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:18.791664 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8r49\" (UniqueName: \"kubernetes.io/projected/e494384c-192a-4c1e-8bd8-5c0389817cc5-kube-api-access-r8r49\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:19.242881 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:19.242848 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-maas-api-tls\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:19.245285 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:19.245249 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-maas-api-tls\") pod \"maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:19.348495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:19.348465 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:19.725983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:19.725955 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9"] Nov 03 19:21:19.726681 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:21:19.726657 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode494384c_192a_4c1e_8bd8_5c0389817cc5.slice/crio-91834e212c7ff97fa1b363941318b8d6f54df81d994dcb4d4bdf5dc4642f80a6 WatchSource:0}: Error finding container 91834e212c7ff97fa1b363941318b8d6f54df81d994dcb4d4bdf5dc4642f80a6: Status 404 returned error can't find the container with id 91834e212c7ff97fa1b363941318b8d6f54df81d994dcb4d4bdf5dc4642f80a6 Nov 03 19:21:19.728231 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:19.728211 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:21:20.297851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:20.297815 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" event={"ID":"e494384c-192a-4c1e-8bd8-5c0389817cc5","Type":"ContainerStarted","Data":"91834e212c7ff97fa1b363941318b8d6f54df81d994dcb4d4bdf5dc4642f80a6"} Nov 03 19:21:21.305710 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:21.305677 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" event={"ID":"e494384c-192a-4c1e-8bd8-5c0389817cc5","Type":"ContainerStarted","Data":"974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971"} Nov 03 19:21:21.306114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:21.305826 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:21.444370 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:21.441382 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" podStartSLOduration=2.513677617 podStartE2EDuration="3.441310044s" totalImagesPullingTime="927.632427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:21:21.439684525 +0000 UTC m=+2215.412245729" watchObservedRunningTime="2026-11-03 19:21:21.441310044 +0000 UTC m=+2215.413871226" Nov 03 19:21:25.457350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:25.453695 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982"] Nov 03 19:21:25.457350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:25.453929 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerName="istio-proxy" containerID="cri-o://b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb" gracePeriod=30 Nov 03 19:21:27.317728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:27.317702 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:21:28.926371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:28.926320 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv"] Nov 03 19:21:28.932107 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:28.932086 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:28.990835 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:28.990802 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d86d1cd0-serving-cert\"" Nov 03 19:21:29.015052 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.015019 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-metrics-certs\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.015224 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.015069 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f4064a-a444-449d-9769-fe2eb909d71f-openshift-service-ca\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.015302 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.015232 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8crhl\" (UniqueName: \"kubernetes.io/projected/33f4064a-a444-449d-9769-fe2eb909d71f-kube-api-access-8crhl\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.015302 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.015274 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-maas-api-tls\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.043548 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.043513 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv"] Nov 03 19:21:29.117761 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.117728 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-metrics-certs\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.118023 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.117998 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f4064a-a444-449d-9769-fe2eb909d71f-openshift-service-ca\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.118283 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.118262 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8crhl\" (UniqueName: \"kubernetes.io/projected/33f4064a-a444-449d-9769-fe2eb909d71f-kube-api-access-8crhl\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.118491 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.118472 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-maas-api-tls\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.119634 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.118800 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f4064a-a444-449d-9769-fe2eb909d71f-openshift-service-ca\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.119735 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:21:29.118876 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-serving-cert: secret "maas-api-e2e-mt-d86d1cd0-serving-cert" not found Nov 03 19:21:29.119735 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:21:29.119721 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-maas-api-tls podName:33f4064a-a444-449d-9769-fe2eb909d71f nodeName:}" failed. No retries permitted until 2026-11-03 19:21:29.619698498 +0000 UTC m=+2223.592259673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-maas-api-tls") pod "maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" (UID: "33f4064a-a444-449d-9769-fe2eb909d71f") : secret "maas-api-e2e-mt-d86d1cd0-serving-cert" not found Nov 03 19:21:29.122082 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.122055 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-metrics-certs\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.178207 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.178114 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8crhl\" (UniqueName: \"kubernetes.io/projected/33f4064a-a444-449d-9769-fe2eb909d71f-kube-api-access-8crhl\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.665952 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.665873 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-maas-api-tls\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.668470 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.668443 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-maas-api-tls\") pod \"maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:29.845980 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:29.845954 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:30.225853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.225817 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv"] Nov 03 19:21:30.226253 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:21:30.225854 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f4064a_a444_449d_9769_fe2eb909d71f.slice/crio-52902967b4e1e7a28ffbb21c6dd69f0dbc5b9350514a7643f5605b58ada8faff WatchSource:0}: Error finding container 52902967b4e1e7a28ffbb21c6dd69f0dbc5b9350514a7643f5605b58ada8faff: Status 404 returned error can't find the container with id 52902967b4e1e7a28ffbb21c6dd69f0dbc5b9350514a7643f5605b58ada8faff Nov 03 19:21:30.557961 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.557927 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" event={"ID":"33f4064a-a444-449d-9769-fe2eb909d71f","Type":"ContainerStarted","Data":"52902967b4e1e7a28ffbb21c6dd69f0dbc5b9350514a7643f5605b58ada8faff"} Nov 03 19:21:30.778151 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.778127 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:21:30.889886 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.889855 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-envoy\") pod \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " Nov 03 19:21:30.890081 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.889914 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-credential-socket\") pod \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " Nov 03 19:21:30.890081 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.890020 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-kube-api-access-dc7sl\" (UniqueName: \"kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-kube-api-access-dc7sl\") pod \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " Nov 03 19:21:30.890081 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.890049 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-token\" (UniqueName: \"kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-token\") pod \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " Nov 03 19:21:30.890262 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.890101 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-certs\") pod \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " Nov 03 19:21:30.890262 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.890151 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-data\") pod \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " Nov 03 19:21:30.890403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.890249 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-credential-socket" pod "c0cdf3bc-673f-4fd1-bb49-92b3e4994229" (UID: "c0cdf3bc-673f-4fd1-bb49-92b3e4994229"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:30.890467 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.890400 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-certs" pod "c0cdf3bc-673f-4fd1-bb49-92b3e4994229" (UID: "c0cdf3bc-673f-4fd1-bb49-92b3e4994229"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:30.890537 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.890510 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-data" pod "c0cdf3bc-673f-4fd1-bb49-92b3e4994229" (UID: "c0cdf3bc-673f-4fd1-bb49-92b3e4994229"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:30.890595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.890546 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-podinfo\") pod \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " Nov 03 19:21:30.890649 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.890638 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istiod-ca-cert\") pod \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " Nov 03 19:21:30.890747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.890722 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-socket\") pod \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\" (UID: \"c0cdf3bc-673f-4fd1-bb49-92b3e4994229\") " Nov 03 19:21:30.891023 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.891001 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istiod-ca-cert" pod "c0cdf3bc-673f-4fd1-bb49-92b3e4994229" (UID: "c0cdf3bc-673f-4fd1-bb49-92b3e4994229"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:21:30.891109 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.891034 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-socket" pod "c0cdf3bc-673f-4fd1-bb49-92b3e4994229" (UID: "c0cdf3bc-673f-4fd1-bb49-92b3e4994229"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:30.891757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.891734 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:30.891810 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.891765 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:30.891810 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.891781 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:30.891810 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.891798 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:30.891905 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.891813 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:30.892492 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.892471 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-token" pod "c0cdf3bc-673f-4fd1-bb49-92b3e4994229" (UID: "c0cdf3bc-673f-4fd1-bb49-92b3e4994229"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:30.892657 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.892633 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-envoy" pod "c0cdf3bc-673f-4fd1-bb49-92b3e4994229" (UID: "c0cdf3bc-673f-4fd1-bb49-92b3e4994229"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:30.892721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.892656 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-kube-api-access-dc7sl" pod "c0cdf3bc-673f-4fd1-bb49-92b3e4994229" (UID: "c0cdf3bc-673f-4fd1-bb49-92b3e4994229"). InnerVolumeSpecName "kube-api-access-dc7sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:30.892721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.892666 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-podinfo" pod "c0cdf3bc-673f-4fd1-bb49-92b3e4994229" (UID: "c0cdf3bc-673f-4fd1-bb49-92b3e4994229"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:21:30.993574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.993499 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc7sl\" (UniqueName: \"kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-kube-api-access-dc7sl\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:30.993574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.993529 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:30.993574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.993540 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:30.993574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:30.993553 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cdf3bc-673f-4fd1-bb49-92b3e4994229-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:31.575087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.566896 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" event={"ID":"33f4064a-a444-449d-9769-fe2eb909d71f","Type":"ContainerStarted","Data":"6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d"} Nov 03 19:21:31.575087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.566962 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:31.575087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.568891 5019 generic.go:356] "Generic (PLEG): container finished" podID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerID="b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb" exitCode=0 Nov 03 19:21:31.575087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.568926 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" Nov 03 19:21:31.575087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.568941 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" event={"ID":"c0cdf3bc-673f-4fd1-bb49-92b3e4994229","Type":"ContainerDied","Data":"b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb"} Nov 03 19:21:31.575087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.568982 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982" event={"ID":"c0cdf3bc-673f-4fd1-bb49-92b3e4994229","Type":"ContainerDied","Data":"fdf2724493cf8d39073c32d5115d94e5c9a17a105e801152cb647454dd6995ab"} Nov 03 19:21:31.575087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.569005 5019 scope.go:122] "RemoveContainer" containerID="b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb" Nov 03 19:21:31.580283 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.580266 5019 scope.go:122] "RemoveContainer" containerID="b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1" Nov 03 19:21:31.590154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.590136 5019 scope.go:122] "RemoveContainer" containerID="b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb" Nov 03 19:21:31.590473 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:21:31.590446 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb\": container with ID starting with b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb not found: ID does not exist" containerID="b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb" Nov 03 19:21:31.590559 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.590483 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb"} err="failed to get container status \"b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb\": rpc error: code = NotFound desc = could not find container \"b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb\": container with ID starting with b4d1fea7552f453094fc2c2f91efee04f94ded369d663f1a958160cf39fe22bb not found: ID does not exist" Nov 03 19:21:31.590559 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.590500 5019 scope.go:122] "RemoveContainer" containerID="b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1" Nov 03 19:21:31.590729 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:21:31.590712 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1\": container with ID starting with b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1 not found: ID does not exist" containerID="b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1" Nov 03 19:21:31.590808 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.590735 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1"} err="failed to get container status \"b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1\": rpc error: code = NotFound desc = could not find container \"b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1\": container with ID starting with b6cb81029a091b0aea91a444607bfc41456107821d888bc487e80c4929e9abd1 not found: ID does not exist" Nov 03 19:21:31.682056 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.681989 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" podStartSLOduration=2.9110783810000003 podStartE2EDuration="3.681971336s" totalImagesPullingTime="770.892955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:21:31.67731494 +0000 UTC m=+2225.649876121" watchObservedRunningTime="2026-11-03 19:21:31.681971336 +0000 UTC m=+2225.654532520" Nov 03 19:21:31.786081 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.786050 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982"] Nov 03 19:21:31.790643 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:31.790617 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-08baeb-openshift-default-748568f9f7-jt982"] Nov 03 19:21:32.152288 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:32.152251 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" path="/var/lib/kubelet/pods/c0cdf3bc-673f-4fd1-bb49-92b3e4994229/volumes" Nov 03 19:21:37.595648 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:37.595617 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:21:38.407733 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:38.407625 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp"] Nov 03 19:21:38.508553 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:38.508516 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc"] Nov 03 19:21:41.298996 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:21:41.298954 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:21:43.408018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.407973 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" podUID="ee9b253d-9da9-4b78-8894-860f9645f1b1" containerName="payload-processing" containerID="cri-o://5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983" gracePeriod=25 Nov 03 19:21:43.509606 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.509549 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" podUID="97ec3cf1-6c12-41b3-9c86-7849471eb005" containerName="payload-pre-processing" containerID="cri-o://bb07809319d7cc160ddd28547f29ab95cf2cac20648e2326ac036d6a43ca6c8b" gracePeriod=25 Nov 03 19:21:43.712782 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.712752 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" Nov 03 19:21:43.743397 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.743358 5019 generic.go:356] "Generic (PLEG): container finished" podID="97ec3cf1-6c12-41b3-9c86-7849471eb005" containerID="bb07809319d7cc160ddd28547f29ab95cf2cac20648e2326ac036d6a43ca6c8b" exitCode=0 Nov 03 19:21:43.743551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.743433 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" event={"ID":"97ec3cf1-6c12-41b3-9c86-7849471eb005","Type":"ContainerDied","Data":"bb07809319d7cc160ddd28547f29ab95cf2cac20648e2326ac036d6a43ca6c8b"} Nov 03 19:21:43.745028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.745004 5019 generic.go:356] "Generic (PLEG): container finished" podID="ee9b253d-9da9-4b78-8894-860f9645f1b1" containerID="5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983" exitCode=0 Nov 03 19:21:43.745135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.745055 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" Nov 03 19:21:43.745135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.745086 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" event={"ID":"ee9b253d-9da9-4b78-8894-860f9645f1b1","Type":"ContainerDied","Data":"5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983"} Nov 03 19:21:43.745206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.745137 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp" event={"ID":"ee9b253d-9da9-4b78-8894-860f9645f1b1","Type":"ContainerDied","Data":"275a4655731105f8ace84a755777d538e500a1702d688839f789f82d8d8f1192"} Nov 03 19:21:43.745206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.745162 5019 scope.go:122] "RemoveContainer" containerID="5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983" Nov 03 19:21:43.759764 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.759736 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9b253d-9da9-4b78-8894-860f9645f1b1-kube-api-access-jbmmx\" (UniqueName: \"kubernetes.io/projected/ee9b253d-9da9-4b78-8894-860f9645f1b1-kube-api-access-jbmmx\") pod \"ee9b253d-9da9-4b78-8894-860f9645f1b1\" (UID: \"ee9b253d-9da9-4b78-8894-860f9645f1b1\") " Nov 03 19:21:43.759906 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.759888 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee9b253d-9da9-4b78-8894-860f9645f1b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9b253d-9da9-4b78-8894-860f9645f1b1-plugins-config-volume\") pod \"ee9b253d-9da9-4b78-8894-860f9645f1b1\" (UID: \"ee9b253d-9da9-4b78-8894-860f9645f1b1\") " Nov 03 19:21:43.761435 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.761041 5019 scope.go:122] "RemoveContainer" containerID="5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983" Nov 03 19:21:43.762681 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:21:43.762644 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983\": container with ID starting with 5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983 not found: ID does not exist" containerID="5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983" Nov 03 19:21:43.762803 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.762683 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983"} err="failed to get container status \"5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983\": rpc error: code = NotFound desc = could not find container \"5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983\": container with ID starting with 5ed4d4946726ef20d3bd3df6ae3ad4ce1c8610d8650070bf3da6dca4f568e983 not found: ID does not exist" Nov 03 19:21:43.762981 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.762954 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9b253d-9da9-4b78-8894-860f9645f1b1-plugins-config-volume" pod "ee9b253d-9da9-4b78-8894-860f9645f1b1" (UID: "ee9b253d-9da9-4b78-8894-860f9645f1b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:21:43.768388 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.765580 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9b253d-9da9-4b78-8894-860f9645f1b1-kube-api-access-jbmmx" pod "ee9b253d-9da9-4b78-8894-860f9645f1b1" (UID: "ee9b253d-9da9-4b78-8894-860f9645f1b1"). InnerVolumeSpecName "kube-api-access-jbmmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:43.832301 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.832277 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" Nov 03 19:21:43.861689 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.861655 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ec3cf1-6c12-41b3-9c86-7849471eb005-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ec3cf1-6c12-41b3-9c86-7849471eb005-plugins-config-volume\") pod \"97ec3cf1-6c12-41b3-9c86-7849471eb005\" (UID: \"97ec3cf1-6c12-41b3-9c86-7849471eb005\") " Nov 03 19:21:43.861875 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.861730 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ec3cf1-6c12-41b3-9c86-7849471eb005-kube-api-access-twv78\" (UniqueName: \"kubernetes.io/projected/97ec3cf1-6c12-41b3-9c86-7849471eb005-kube-api-access-twv78\") pod \"97ec3cf1-6c12-41b3-9c86-7849471eb005\" (UID: \"97ec3cf1-6c12-41b3-9c86-7849471eb005\") " Nov 03 19:21:43.862568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.862320 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ec3cf1-6c12-41b3-9c86-7849471eb005-plugins-config-volume" pod "97ec3cf1-6c12-41b3-9c86-7849471eb005" (UID: "97ec3cf1-6c12-41b3-9c86-7849471eb005"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:21:43.863781 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.863753 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbmmx\" (UniqueName: \"kubernetes.io/projected/ee9b253d-9da9-4b78-8894-860f9645f1b1-kube-api-access-jbmmx\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:43.863912 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.863783 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9b253d-9da9-4b78-8894-860f9645f1b1-plugins-config-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:43.863912 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.863799 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ec3cf1-6c12-41b3-9c86-7849471eb005-plugins-config-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:43.865302 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.865276 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ec3cf1-6c12-41b3-9c86-7849471eb005-kube-api-access-twv78" pod "97ec3cf1-6c12-41b3-9c86-7849471eb005" (UID: "97ec3cf1-6c12-41b3-9c86-7849471eb005"). InnerVolumeSpecName "kube-api-access-twv78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:43.966142 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:43.966068 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twv78\" (UniqueName: \"kubernetes.io/projected/97ec3cf1-6c12-41b3-9c86-7849471eb005-kube-api-access-twv78\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:44.166231 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:44.166201 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp"] Nov 03 19:21:44.173022 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:44.172995 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dd1327-764f66ccbb-stnrp"] Nov 03 19:21:44.752904 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:44.752804 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" event={"ID":"97ec3cf1-6c12-41b3-9c86-7849471eb005","Type":"ContainerDied","Data":"65f1dfd041b6c6ad6c9e2a101dbaa0855bef5b21f0d7e54f242f0b5711567470"} Nov 03 19:21:44.752904 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:44.752850 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc" Nov 03 19:21:44.752904 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:44.752881 5019 scope.go:122] "RemoveContainer" containerID="bb07809319d7cc160ddd28547f29ab95cf2cac20648e2326ac036d6a43ca6c8b" Nov 03 19:21:44.911379 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:44.911339 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc"] Nov 03 19:21:44.956089 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:44.956054 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1327-5bbfddcc47-zjbnc"] Nov 03 19:21:46.151068 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:46.151035 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ec3cf1-6c12-41b3-9c86-7849471eb005" path="/var/lib/kubelet/pods/97ec3cf1-6c12-41b3-9c86-7849471eb005/volumes" Nov 03 19:21:46.151491 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:21:46.151476 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee9b253d-9da9-4b78-8894-860f9645f1b1" path="/var/lib/kubelet/pods/ee9b253d-9da9-4b78-8894-860f9645f1b1/volumes" Nov 03 19:22:02.414540 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.414505 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5"] Nov 03 19:22:02.415153 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.414705 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" podUID="d508cbeb-a7cf-464f-88d4-b3374365e690" containerName="maas-api" containerID="cri-o://0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b" gracePeriod=30 Nov 03 19:22:02.561569 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.561525 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" podUID="d508cbeb-a7cf-464f-88d4-b3374365e690" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.119:8443/health\": dial tcp 10.132.0.119:8443: connect: connection refused" Nov 03 19:22:02.782687 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.782658 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:22:02.893757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.891615 5019 generic.go:356] "Generic (PLEG): container finished" podID="d508cbeb-a7cf-464f-88d4-b3374365e690" containerID="0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b" exitCode=0 Nov 03 19:22:02.893757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.891706 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" event={"ID":"d508cbeb-a7cf-464f-88d4-b3374365e690","Type":"ContainerDied","Data":"0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b"} Nov 03 19:22:02.893757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.891754 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" event={"ID":"d508cbeb-a7cf-464f-88d4-b3374365e690","Type":"ContainerDied","Data":"36cda8e186eeb6d78a6517e766326dfd501700ccc8ba7828c313161d28a5904a"} Nov 03 19:22:02.893757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.891785 5019 scope.go:122] "RemoveContainer" containerID="0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b" Nov 03 19:22:02.893757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.891957 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5" Nov 03 19:22:02.900765 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.899369 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d508cbeb-a7cf-464f-88d4-b3374365e690-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d508cbeb-a7cf-464f-88d4-b3374365e690-openshift-service-ca\") pod \"d508cbeb-a7cf-464f-88d4-b3374365e690\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " Nov 03 19:22:02.900765 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.899593 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d508cbeb-a7cf-464f-88d4-b3374365e690-kube-api-access-n82rn\" (UniqueName: \"kubernetes.io/projected/d508cbeb-a7cf-464f-88d4-b3374365e690-kube-api-access-n82rn\") pod \"d508cbeb-a7cf-464f-88d4-b3374365e690\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " Nov 03 19:22:02.900765 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.899713 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-maas-api-tls\") pod \"d508cbeb-a7cf-464f-88d4-b3374365e690\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " Nov 03 19:22:02.900765 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.899763 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-metrics-certs\") pod \"d508cbeb-a7cf-464f-88d4-b3374365e690\" (UID: \"d508cbeb-a7cf-464f-88d4-b3374365e690\") " Nov 03 19:22:02.902395 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.902365 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d508cbeb-a7cf-464f-88d4-b3374365e690-openshift-service-ca" pod "d508cbeb-a7cf-464f-88d4-b3374365e690" (UID: "d508cbeb-a7cf-464f-88d4-b3374365e690"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:22:02.917489 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.909628 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-metrics-certs" pod "d508cbeb-a7cf-464f-88d4-b3374365e690" (UID: "d508cbeb-a7cf-464f-88d4-b3374365e690"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:22:02.917489 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.911131 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d508cbeb-a7cf-464f-88d4-b3374365e690-kube-api-access-n82rn" pod "d508cbeb-a7cf-464f-88d4-b3374365e690" (UID: "d508cbeb-a7cf-464f-88d4-b3374365e690"). InnerVolumeSpecName "kube-api-access-n82rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:02.917489 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.912583 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-maas-api-tls" pod "d508cbeb-a7cf-464f-88d4-b3374365e690" (UID: "d508cbeb-a7cf-464f-88d4-b3374365e690"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:22:02.945727 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.945682 5019 scope.go:122] "RemoveContainer" containerID="0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b" Nov 03 19:22:02.946679 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:02.946645 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b\": container with ID starting with 0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b not found: ID does not exist" containerID="0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b" Nov 03 19:22:02.946792 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:02.946689 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b"} err="failed to get container status \"0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b\": rpc error: code = NotFound desc = could not find container \"0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b\": container with ID starting with 0cb391f43dc3707cc0f47750e8dac0f42ba35bc73bbbd1265cdba6399086469b not found: ID does not exist" Nov 03 19:22:03.001690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:03.001605 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n82rn\" (UniqueName: \"kubernetes.io/projected/d508cbeb-a7cf-464f-88d4-b3374365e690-kube-api-access-n82rn\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.001690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:03.001638 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.001690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:03.001649 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d508cbeb-a7cf-464f-88d4-b3374365e690-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.001690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:03.001658 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d508cbeb-a7cf-464f-88d4-b3374365e690-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.349405 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:03.337674 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5"] Nov 03 19:22:03.350147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:03.350115 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bbccf3-b669bbc49-w6rl5"] Nov 03 19:22:04.151617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:04.151587 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d508cbeb-a7cf-464f-88d4-b3374365e690" path="/var/lib/kubelet/pods/d508cbeb-a7cf-464f-88d4-b3374365e690/volumes" Nov 03 19:22:04.831056 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:04.831020 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc"] Nov 03 19:22:04.831299 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:04.831267 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerName="istio-proxy" containerID="cri-o://a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6" gracePeriod=30 Nov 03 19:22:10.129147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.129119 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 19:22:10.205092 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205058 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-credential-socket\") pod \"8aa2f0da-f709-4e28-a544-e7ff20f78633\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " Nov 03 19:22:10.205092 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205096 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-socket\") pod \"8aa2f0da-f709-4e28-a544-e7ff20f78633\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " Nov 03 19:22:10.205289 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205155 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-token\" (UniqueName: \"kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-token\") pod \"8aa2f0da-f709-4e28-a544-e7ff20f78633\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " Nov 03 19:22:10.205289 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205262 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-certs\") pod \"8aa2f0da-f709-4e28-a544-e7ff20f78633\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " Nov 03 19:22:10.205415 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205363 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-credential-socket" pod "8aa2f0da-f709-4e28-a544-e7ff20f78633" (UID: "8aa2f0da-f709-4e28-a544-e7ff20f78633"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:10.205479 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205444 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-kube-api-access-589zn\" (UniqueName: \"kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-kube-api-access-589zn\") pod \"8aa2f0da-f709-4e28-a544-e7ff20f78633\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " Nov 03 19:22:10.205479 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205445 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-socket" pod "8aa2f0da-f709-4e28-a544-e7ff20f78633" (UID: "8aa2f0da-f709-4e28-a544-e7ff20f78633"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:10.205591 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205504 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-certs" pod "8aa2f0da-f709-4e28-a544-e7ff20f78633" (UID: "8aa2f0da-f709-4e28-a544-e7ff20f78633"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:10.205591 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205561 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-envoy\") pod \"8aa2f0da-f709-4e28-a544-e7ff20f78633\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " Nov 03 19:22:10.205704 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205594 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-data\") pod \"8aa2f0da-f709-4e28-a544-e7ff20f78633\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " Nov 03 19:22:10.205704 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205657 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-podinfo\") pod \"8aa2f0da-f709-4e28-a544-e7ff20f78633\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " Nov 03 19:22:10.205704 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205686 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa2f0da-f709-4e28-a544-e7ff20f78633-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa2f0da-f709-4e28-a544-e7ff20f78633-istiod-ca-cert\") pod \"8aa2f0da-f709-4e28-a544-e7ff20f78633\" (UID: \"8aa2f0da-f709-4e28-a544-e7ff20f78633\") " Nov 03 19:22:10.205932 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.205903 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-data" pod "8aa2f0da-f709-4e28-a544-e7ff20f78633" (UID: "8aa2f0da-f709-4e28-a544-e7ff20f78633"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:10.206076 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.206053 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa2f0da-f709-4e28-a544-e7ff20f78633-istiod-ca-cert" pod "8aa2f0da-f709-4e28-a544-e7ff20f78633" (UID: "8aa2f0da-f709-4e28-a544-e7ff20f78633"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:22:10.206410 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.206390 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:10.206497 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.206418 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:10.206497 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.206434 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa2f0da-f709-4e28-a544-e7ff20f78633-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:10.206497 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.206450 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:10.206497 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.206465 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:10.207813 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.207776 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-podinfo" pod "8aa2f0da-f709-4e28-a544-e7ff20f78633" (UID: "8aa2f0da-f709-4e28-a544-e7ff20f78633"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:22:10.207912 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.207787 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-kube-api-access-589zn" pod "8aa2f0da-f709-4e28-a544-e7ff20f78633" (UID: "8aa2f0da-f709-4e28-a544-e7ff20f78633"). InnerVolumeSpecName "kube-api-access-589zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:10.207912 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.207884 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-envoy" pod "8aa2f0da-f709-4e28-a544-e7ff20f78633" (UID: "8aa2f0da-f709-4e28-a544-e7ff20f78633"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:10.207912 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.207892 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-token" pod "8aa2f0da-f709-4e28-a544-e7ff20f78633" (UID: "8aa2f0da-f709-4e28-a544-e7ff20f78633"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:10.308766 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.308726 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:10.308933 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.308771 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:10.308933 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.308788 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-589zn\" (UniqueName: \"kubernetes.io/projected/8aa2f0da-f709-4e28-a544-e7ff20f78633-kube-api-access-589zn\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:10.308933 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.308803 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa2f0da-f709-4e28-a544-e7ff20f78633-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:10.951419 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.951382 5019 generic.go:356] "Generic (PLEG): container finished" podID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerID="a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6" exitCode=0 Nov 03 19:22:10.951585 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.951432 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" Nov 03 19:22:10.951585 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.951463 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" event={"ID":"8aa2f0da-f709-4e28-a544-e7ff20f78633","Type":"ContainerDied","Data":"a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6"} Nov 03 19:22:10.951585 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.951522 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc" event={"ID":"8aa2f0da-f709-4e28-a544-e7ff20f78633","Type":"ContainerDied","Data":"001e5fd832ee9e3003d0bdade7ce2533395d4798c0e390df7fe487e5fa961daf"} Nov 03 19:22:10.951585 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.951554 5019 scope.go:122] "RemoveContainer" containerID="a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6" Nov 03 19:22:10.963914 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.963899 5019 scope.go:122] "RemoveContainer" containerID="20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70" Nov 03 19:22:10.975058 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.975029 5019 scope.go:122] "RemoveContainer" containerID="a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6" Nov 03 19:22:10.975400 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:10.975352 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6\": container with ID starting with a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6 not found: ID does not exist" containerID="a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6" Nov 03 19:22:10.975479 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.975407 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6"} err="failed to get container status \"a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6\": rpc error: code = NotFound desc = could not find container \"a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6\": container with ID starting with a2b550fd48838b0d7dfe6f5c631a7dfd042d46ae2ff7124efdf9ecc1c4e56dc6 not found: ID does not exist" Nov 03 19:22:10.975479 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.975427 5019 scope.go:122] "RemoveContainer" containerID="20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70" Nov 03 19:22:10.975710 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:10.975693 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70\": container with ID starting with 20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70 not found: ID does not exist" containerID="20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70" Nov 03 19:22:10.975752 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:10.975717 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70"} err="failed to get container status \"20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70\": rpc error: code = NotFound desc = could not find container \"20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70\": container with ID starting with 20c799f7be34514ace2441a0548a542118e6fbc015a2537fe31e0ab241752b70 not found: ID does not exist" Nov 03 19:22:11.157585 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:11.157550 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc"] Nov 03 19:22:11.161950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:11.161923 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-gbnrc"] Nov 03 19:22:11.317180 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:11.317095 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:22:11.317492 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:11.317465 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb"] Nov 03 19:22:11.317750 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:11.317721 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" containerID="cri-o://46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d" gracePeriod=30 Nov 03 19:22:12.151813 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:12.151780 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" path="/var/lib/kubelet/pods/8aa2f0da-f709-4e28-a544-e7ff20f78633/volumes" Nov 03 19:22:15.347600 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:15.347555 5019 patch_prober.go:29] interesting pod/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:22:15.348108 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:15.347640 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:22:16.616248 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.616220 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:22:16.702408 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.702367 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-data\") pod \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " Nov 03 19:22:16.702579 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.702435 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-podinfo\") pod \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " Nov 03 19:22:16.702579 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.702518 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istiod-ca-cert\") pod \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " Nov 03 19:22:16.702654 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.702575 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-data" pod "dd9a1357-9f0e-4616-8e9a-6c60f4385cec" (UID: "dd9a1357-9f0e-4616-8e9a-6c60f4385cec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:16.702654 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.702623 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-envoy\") pod \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " Nov 03 19:22:16.702772 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.702677 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-kube-api-access-rztvl\" (UniqueName: \"kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-kube-api-access-rztvl\") pod \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " Nov 03 19:22:16.702772 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.702759 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-credential-socket\") pod \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " Nov 03 19:22:16.702889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.702811 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-token\" (UniqueName: \"kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-token\") pod \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " Nov 03 19:22:16.702889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.702882 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-socket\") pod \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " Nov 03 19:22:16.702999 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.702894 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istiod-ca-cert" pod "dd9a1357-9f0e-4616-8e9a-6c60f4385cec" (UID: "dd9a1357-9f0e-4616-8e9a-6c60f4385cec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:22:16.702999 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.702980 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-credential-socket" pod "dd9a1357-9f0e-4616-8e9a-6c60f4385cec" (UID: "dd9a1357-9f0e-4616-8e9a-6c60f4385cec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:16.703111 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.703001 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-certs\") pod \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\" (UID: \"dd9a1357-9f0e-4616-8e9a-6c60f4385cec\") " Nov 03 19:22:16.703172 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.703148 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-socket" pod "dd9a1357-9f0e-4616-8e9a-6c60f4385cec" (UID: "dd9a1357-9f0e-4616-8e9a-6c60f4385cec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:16.703318 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.703295 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-certs" pod "dd9a1357-9f0e-4616-8e9a-6c60f4385cec" (UID: "dd9a1357-9f0e-4616-8e9a-6c60f4385cec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:16.703677 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.703655 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:16.703736 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.703684 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:16.703736 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.703700 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:16.703736 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.703715 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:16.703736 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.703731 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:16.704772 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.704746 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-podinfo" pod "dd9a1357-9f0e-4616-8e9a-6c60f4385cec" (UID: "dd9a1357-9f0e-4616-8e9a-6c60f4385cec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:22:16.705053 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.705032 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-token" pod "dd9a1357-9f0e-4616-8e9a-6c60f4385cec" (UID: "dd9a1357-9f0e-4616-8e9a-6c60f4385cec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:16.705228 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.705202 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-envoy" pod "dd9a1357-9f0e-4616-8e9a-6c60f4385cec" (UID: "dd9a1357-9f0e-4616-8e9a-6c60f4385cec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:16.705228 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.705208 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-kube-api-access-rztvl" pod "dd9a1357-9f0e-4616-8e9a-6c60f4385cec" (UID: "dd9a1357-9f0e-4616-8e9a-6c60f4385cec"). InnerVolumeSpecName "kube-api-access-rztvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:16.804749 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.804711 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:16.804749 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.804740 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:16.804749 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.804756 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rztvl\" (UniqueName: \"kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-kube-api-access-rztvl\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:16.804986 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.804768 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9a1357-9f0e-4616-8e9a-6c60f4385cec-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:16.997708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.997670 5019 generic.go:356] "Generic (PLEG): container finished" podID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerID="46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d" exitCode=0 Nov 03 19:22:16.997885 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.997709 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" event={"ID":"dd9a1357-9f0e-4616-8e9a-6c60f4385cec","Type":"ContainerDied","Data":"46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d"} Nov 03 19:22:16.997885 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.997727 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" Nov 03 19:22:16.997885 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.997772 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb" event={"ID":"dd9a1357-9f0e-4616-8e9a-6c60f4385cec","Type":"ContainerDied","Data":"aa85916419b876d9301dae69ee7b635999394cde13e7b9f14ab55e6b9c5f2c28"} Nov 03 19:22:16.997885 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:16.997796 5019 scope.go:122] "RemoveContainer" containerID="46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d" Nov 03 19:22:17.009270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:17.009252 5019 scope.go:122] "RemoveContainer" containerID="3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e" Nov 03 19:22:17.018614 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:17.018598 5019 scope.go:122] "RemoveContainer" containerID="46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d" Nov 03 19:22:17.018865 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:17.018841 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d\": container with ID starting with 46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d not found: ID does not exist" containerID="46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d" Nov 03 19:22:17.018915 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:17.018873 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d"} err="failed to get container status \"46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d\": rpc error: code = NotFound desc = could not find container \"46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d\": container with ID starting with 46a033da51f29fd242fde30665752607731a14e0ba948ceff6697138a32ce44d not found: ID does not exist" Nov 03 19:22:17.018915 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:17.018893 5019 scope.go:122] "RemoveContainer" containerID="3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e" Nov 03 19:22:17.019135 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:17.019118 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e\": container with ID starting with 3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e not found: ID does not exist" containerID="3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e" Nov 03 19:22:17.019187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:17.019138 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e"} err="failed to get container status \"3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e\": rpc error: code = NotFound desc = could not find container \"3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e\": container with ID starting with 3b0b71af2c9b0bfe203b1ee2cf19ac7b985a0ee408c694f5279abed43cc25c6e not found: ID does not exist" Nov 03 19:22:17.154212 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:17.154178 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb"] Nov 03 19:22:17.157981 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:17.157955 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bbccf3-openshift-default-546f969587-8pgfb"] Nov 03 19:22:18.152690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:18.152655 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" path="/var/lib/kubelet/pods/dd9a1357-9f0e-4616-8e9a-6c60f4385cec/volumes" Nov 03 19:22:23.649320 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:23.649284 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk"] Nov 03 19:22:28.650585 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:28.650541 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" podUID="7373d851-49e8-4249-bd94-2a5c95e3d625" containerName="payload-pre-processing" containerID="cri-o://8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37" gracePeriod=25 Nov 03 19:22:28.939252 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:28.938872 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" Nov 03 19:22:29.084452 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.084420 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7373d851-49e8-4249-bd94-2a5c95e3d625-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7373d851-49e8-4249-bd94-2a5c95e3d625-plugins-config-volume\") pod \"7373d851-49e8-4249-bd94-2a5c95e3d625\" (UID: \"7373d851-49e8-4249-bd94-2a5c95e3d625\") " Nov 03 19:22:29.084747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.084727 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7373d851-49e8-4249-bd94-2a5c95e3d625-kube-api-access-mdznk\" (UniqueName: \"kubernetes.io/projected/7373d851-49e8-4249-bd94-2a5c95e3d625-kube-api-access-mdznk\") pod \"7373d851-49e8-4249-bd94-2a5c95e3d625\" (UID: \"7373d851-49e8-4249-bd94-2a5c95e3d625\") " Nov 03 19:22:29.084922 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.084889 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7373d851-49e8-4249-bd94-2a5c95e3d625-plugins-config-volume" pod "7373d851-49e8-4249-bd94-2a5c95e3d625" (UID: "7373d851-49e8-4249-bd94-2a5c95e3d625"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:22:29.085367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.085345 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7373d851-49e8-4249-bd94-2a5c95e3d625-plugins-config-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:29.090584 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.090559 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7373d851-49e8-4249-bd94-2a5c95e3d625-kube-api-access-mdznk" pod "7373d851-49e8-4249-bd94-2a5c95e3d625" (UID: "7373d851-49e8-4249-bd94-2a5c95e3d625"). InnerVolumeSpecName "kube-api-access-mdznk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:29.099838 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.099806 5019 generic.go:356] "Generic (PLEG): container finished" podID="7373d851-49e8-4249-bd94-2a5c95e3d625" containerID="8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37" exitCode=0 Nov 03 19:22:29.099956 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.099919 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" event={"ID":"7373d851-49e8-4249-bd94-2a5c95e3d625","Type":"ContainerDied","Data":"8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37"} Nov 03 19:22:29.100035 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.099966 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" event={"ID":"7373d851-49e8-4249-bd94-2a5c95e3d625","Type":"ContainerDied","Data":"15fb68c90f32ed338d8c944b4767817f12a5ea5bed6158aefec57fa88c8608bf"} Nov 03 19:22:29.100035 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.099992 5019 scope.go:122] "RemoveContainer" containerID="8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37" Nov 03 19:22:29.100158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.100143 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk" Nov 03 19:22:29.114044 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.114024 5019 scope.go:122] "RemoveContainer" containerID="8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37" Nov 03 19:22:29.114341 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:29.114307 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37\": container with ID starting with 8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37 not found: ID does not exist" containerID="8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37" Nov 03 19:22:29.114400 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.114353 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37"} err="failed to get container status \"8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37\": rpc error: code = NotFound desc = could not find container \"8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37\": container with ID starting with 8e28d88019b29290e43f3a5e7b7b3ab0dcd72bb9bb80008c6f9f732dc8d0dc37 not found: ID does not exist" Nov 03 19:22:29.186959 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.186929 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdznk\" (UniqueName: \"kubernetes.io/projected/7373d851-49e8-4249-bd94-2a5c95e3d625-kube-api-access-mdznk\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:29.256939 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.256905 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk"] Nov 03 19:22:29.261211 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:29.261184 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cc3f2-67b59f5f75-b9dfk"] Nov 03 19:22:30.151685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:30.151614 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7373d851-49e8-4249-bd94-2a5c95e3d625" path="/var/lib/kubelet/pods/7373d851-49e8-4249-bd94-2a5c95e3d625/volumes" Nov 03 19:22:32.125966 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.125928 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf"] Nov 03 19:22:32.126747 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126726 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerName="istio-proxy" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126749 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerName="istio-proxy" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126763 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7373d851-49e8-4249-bd94-2a5c95e3d625" containerName="payload-pre-processing" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126768 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7373d851-49e8-4249-bd94-2a5c95e3d625" containerName="payload-pre-processing" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126780 5019 cpu_manager.go:414] "containerMap: removing container" podUID="97ec3cf1-6c12-41b3-9c86-7849471eb005" containerName="payload-pre-processing" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126785 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ec3cf1-6c12-41b3-9c86-7849471eb005" containerName="payload-pre-processing" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126794 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126799 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126806 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126810 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126822 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d508cbeb-a7cf-464f-88d4-b3374365e690" containerName="maas-api" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126828 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d508cbeb-a7cf-464f-88d4-b3374365e690" containerName="maas-api" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126841 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerName="istio-proxy" Nov 03 19:22:32.126843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126846 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerName="istio-proxy" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126868 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ee9b253d-9da9-4b78-8894-860f9645f1b1" containerName="payload-processing" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126872 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee9b253d-9da9-4b78-8894-860f9645f1b1" containerName="payload-processing" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126879 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerName="istio-proxy" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126884 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerName="istio-proxy" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126893 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerName="istio-proxy" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126897 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerName="istio-proxy" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.126991 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.127001 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9a1357-9f0e-4616-8e9a-6c60f4385cec" containerName="istio-proxy" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.127011 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d508cbeb-a7cf-464f-88d4-b3374365e690" containerName="maas-api" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.127016 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerName="istio-proxy" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.127024 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee9b253d-9da9-4b78-8894-860f9645f1b1" containerName="payload-processing" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.127029 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0cdf3bc-673f-4fd1-bb49-92b3e4994229" containerName="istio-proxy" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.127036 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerName="istio-proxy" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.127045 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7373d851-49e8-4249-bd94-2a5c95e3d625" containerName="payload-pre-processing" Nov 03 19:22:32.127229 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.127052 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ec3cf1-6c12-41b3-9c86-7849471eb005" containerName="payload-pre-processing" Nov 03 19:22:32.132457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.132439 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.181257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.181233 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6413ba-serving-cert\"" Nov 03 19:22:32.227988 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.227939 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4773c82-6fb1-4108-aa54-a6e5635a847b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.228206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.228182 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-metrics-certs\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.228454 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.228425 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2vt2\" (UniqueName: \"kubernetes.io/projected/f4773c82-6fb1-4108-aa54-a6e5635a847b-kube-api-access-r2vt2\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.228651 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.228627 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-maas-api-tls\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.231672 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.231645 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf"] Nov 03 19:22:32.329985 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.329949 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-metrics-certs\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.330148 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.330102 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2vt2\" (UniqueName: \"kubernetes.io/projected/f4773c82-6fb1-4108-aa54-a6e5635a847b-kube-api-access-r2vt2\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.330211 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.330177 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-maas-api-tls\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.330301 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:32.330283 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-serving-cert: secret "maas-api-e2e-inf-a-6413ba-serving-cert" not found Nov 03 19:22:32.330422 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:32.330373 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-maas-api-tls podName:f4773c82-6fb1-4108-aa54-a6e5635a847b nodeName:}" failed. No retries permitted until 2026-11-03 19:22:32.830351301 +0000 UTC m=+2286.802912484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-maas-api-tls") pod "maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" (UID: "f4773c82-6fb1-4108-aa54-a6e5635a847b") : secret "maas-api-e2e-inf-a-6413ba-serving-cert" not found Nov 03 19:22:32.330422 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.330395 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4773c82-6fb1-4108-aa54-a6e5635a847b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.331110 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.331078 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4773c82-6fb1-4108-aa54-a6e5635a847b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.332556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.332533 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-metrics-certs\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.390091 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.390027 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2vt2\" (UniqueName: \"kubernetes.io/projected/f4773c82-6fb1-4108-aa54-a6e5635a847b-kube-api-access-r2vt2\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.841361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.841299 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-maas-api-tls\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:32.843848 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:32.843824 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-maas-api-tls\") pod \"maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:33.047138 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:33.047106 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:33.420362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:33.420306 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf"] Nov 03 19:22:33.420733 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:22:33.420467 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4773c82_6fb1_4108_aa54_a6e5635a847b.slice/crio-d035828d87822409539aea10195c2168f5c7a46faff0146be4a2f5757b2280f9 WatchSource:0}: Error finding container d035828d87822409539aea10195c2168f5c7a46faff0146be4a2f5757b2280f9: Status 404 returned error can't find the container with id d035828d87822409539aea10195c2168f5c7a46faff0146be4a2f5757b2280f9 Nov 03 19:22:34.153479 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:34.153445 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" event={"ID":"f4773c82-6fb1-4108-aa54-a6e5635a847b","Type":"ContainerStarted","Data":"d035828d87822409539aea10195c2168f5c7a46faff0146be4a2f5757b2280f9"} Nov 03 19:22:35.157353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:35.157290 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" event={"ID":"f4773c82-6fb1-4108-aa54-a6e5635a847b","Type":"ContainerStarted","Data":"97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023"} Nov 03 19:22:35.157845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:35.157372 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:35.269740 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:35.269675 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" podStartSLOduration=2.330348894 podStartE2EDuration="3.269657103s" totalImagesPullingTime="939.308209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:22:35.266639455 +0000 UTC m=+2289.239200636" watchObservedRunningTime="2026-11-03 19:22:35.269657103 +0000 UTC m=+2289.242218284" Nov 03 19:22:41.169775 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.169689 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:22:41.298306 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:41.298270 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:22:41.504887 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.504796 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns"] Nov 03 19:22:41.505727 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.505704 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa2f0da-f709-4e28-a544-e7ff20f78633" containerName="istio-proxy" Nov 03 19:22:41.509655 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.509637 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:41.558699 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.558674 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-332dac-serving-cert\"" Nov 03 19:22:41.569905 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.569877 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns"] Nov 03 19:22:41.667789 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.667757 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-metrics-certs\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:41.667976 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.667939 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:41.668041 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.667980 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tn58\" (UniqueName: \"kubernetes.io/projected/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-kube-api-access-9tn58\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:41.668139 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.668112 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-maas-api-tls\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:41.774311 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.774209 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-maas-api-tls\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:41.774881 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.774855 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-metrics-certs\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:41.775655 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.775634 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:41.776594 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.776573 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tn58\" (UniqueName: \"kubernetes.io/projected/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-kube-api-access-9tn58\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:41.792084 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:41.774672 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-serving-cert: secret "maas-api-e2e-inf-b-332dac-serving-cert" not found Nov 03 19:22:41.792522 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:41.792499 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-maas-api-tls podName:a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf nodeName:}" failed. No retries permitted until 2026-11-03 19:22:42.292473143 +0000 UTC m=+2296.265034320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-maas-api-tls") pod "maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" (UID: "a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf") : secret "maas-api-e2e-inf-b-332dac-serving-cert" not found Nov 03 19:22:41.793020 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.776498 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:41.818193 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.818160 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-metrics-certs\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:41.849320 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:41.849291 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tn58\" (UniqueName: \"kubernetes.io/projected/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-kube-api-access-9tn58\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:42.298261 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:42.298221 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-maas-api-tls\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:42.300916 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:42.300885 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-maas-api-tls\") pod \"maas-api-e2e-inf-b-332dac-8dffdd8db-74pns\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:42.424211 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:42.424178 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:42.803685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:42.803651 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns"] Nov 03 19:22:42.803685 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:22:42.803664 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda579a5f2_a4c4_4152_b71a_b4a35fc8ccdf.slice/crio-f9188ca34ddc7a1c51ef72f6a07d3c2d67f319c4cdadddff23ec231582b21561 WatchSource:0}: Error finding container f9188ca34ddc7a1c51ef72f6a07d3c2d67f319c4cdadddff23ec231582b21561: Status 404 returned error can't find the container with id f9188ca34ddc7a1c51ef72f6a07d3c2d67f319c4cdadddff23ec231582b21561 Nov 03 19:22:43.216990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:43.216956 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" event={"ID":"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf","Type":"ContainerStarted","Data":"f9188ca34ddc7a1c51ef72f6a07d3c2d67f319c4cdadddff23ec231582b21561"} Nov 03 19:22:44.227178 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:44.227139 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" event={"ID":"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf","Type":"ContainerStarted","Data":"130dd540b979ba3ea22b855ec48bb7ed17d56d4eac326d7b4c3c8620b867e289"} Nov 03 19:22:44.227646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:44.227209 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:44.339698 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:44.339647 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" podStartSLOduration=2.490870031 podStartE2EDuration="3.339632401s" totalImagesPullingTime="848.76237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:22:44.337403375 +0000 UTC m=+2298.309964558" watchObservedRunningTime="2026-11-03 19:22:44.339632401 +0000 UTC m=+2298.312193603" Nov 03 19:22:50.239342 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:50.239306 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:22:55.402829 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:55.402792 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c"] Nov 03 19:22:55.403715 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:55.403679 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" podUID="9b363559-7b87-4dde-a95b-8fe222208696" containerName="maas-api" containerID="cri-o://c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d" gracePeriod=30 Nov 03 19:22:55.930890 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:55.930863 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:22:56.134110 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.134074 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-metrics-certs\") pod \"9b363559-7b87-4dde-a95b-8fe222208696\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " Nov 03 19:22:56.134300 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.134149 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b363559-7b87-4dde-a95b-8fe222208696-kube-api-access-2qv44\" (UniqueName: \"kubernetes.io/projected/9b363559-7b87-4dde-a95b-8fe222208696-kube-api-access-2qv44\") pod \"9b363559-7b87-4dde-a95b-8fe222208696\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " Nov 03 19:22:56.134300 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.134234 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b363559-7b87-4dde-a95b-8fe222208696-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b363559-7b87-4dde-a95b-8fe222208696-openshift-service-ca\") pod \"9b363559-7b87-4dde-a95b-8fe222208696\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " Nov 03 19:22:56.134491 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.134309 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-maas-api-tls\") pod \"9b363559-7b87-4dde-a95b-8fe222208696\" (UID: \"9b363559-7b87-4dde-a95b-8fe222208696\") " Nov 03 19:22:56.134662 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.134637 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b363559-7b87-4dde-a95b-8fe222208696-openshift-service-ca" pod "9b363559-7b87-4dde-a95b-8fe222208696" (UID: "9b363559-7b87-4dde-a95b-8fe222208696"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:22:56.134872 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.134849 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b363559-7b87-4dde-a95b-8fe222208696-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:56.136378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.136352 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-metrics-certs" pod "9b363559-7b87-4dde-a95b-8fe222208696" (UID: "9b363559-7b87-4dde-a95b-8fe222208696"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:22:56.136543 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.136390 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-maas-api-tls" pod "9b363559-7b87-4dde-a95b-8fe222208696" (UID: "9b363559-7b87-4dde-a95b-8fe222208696"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:22:56.136543 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.136426 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b363559-7b87-4dde-a95b-8fe222208696-kube-api-access-2qv44" pod "9b363559-7b87-4dde-a95b-8fe222208696" (UID: "9b363559-7b87-4dde-a95b-8fe222208696"). InnerVolumeSpecName "kube-api-access-2qv44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:56.235577 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.235497 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qv44\" (UniqueName: \"kubernetes.io/projected/9b363559-7b87-4dde-a95b-8fe222208696-kube-api-access-2qv44\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:56.235577 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.235525 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:56.235577 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.235534 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b363559-7b87-4dde-a95b-8fe222208696-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:56.427532 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.427485 5019 generic.go:356] "Generic (PLEG): container finished" podID="9b363559-7b87-4dde-a95b-8fe222208696" containerID="c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d" exitCode=0 Nov 03 19:22:56.428045 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.427661 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" event={"ID":"9b363559-7b87-4dde-a95b-8fe222208696","Type":"ContainerDied","Data":"c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d"} Nov 03 19:22:56.428045 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.427711 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" event={"ID":"9b363559-7b87-4dde-a95b-8fe222208696","Type":"ContainerDied","Data":"7c4461a5ff378734e013ff40814056b3050247e6e392c3a40a8ab3258e9c46fd"} Nov 03 19:22:56.428045 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.427737 5019 scope.go:122] "RemoveContainer" containerID="c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d" Nov 03 19:22:56.428045 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.427904 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c" Nov 03 19:22:56.449838 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.449812 5019 scope.go:122] "RemoveContainer" containerID="c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d" Nov 03 19:22:56.450185 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:22:56.450162 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d\": container with ID starting with c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d not found: ID does not exist" containerID="c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d" Nov 03 19:22:56.450273 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.450194 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d"} err="failed to get container status \"c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d\": rpc error: code = NotFound desc = could not find container \"c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d\": container with ID starting with c2f7ce910b5b6dbccb555421ce02b5302791c0040b5b6c7819d05ae1f0aa1d4d not found: ID does not exist" Nov 03 19:22:56.585963 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.585877 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c"] Nov 03 19:22:56.630147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:56.630112 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-81a044-dd96d8b74-rs26c"] Nov 03 19:22:57.316087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.316047 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg"] Nov 03 19:22:57.316813 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.316790 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9b363559-7b87-4dde-a95b-8fe222208696" containerName="maas-api" Nov 03 19:22:57.316813 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.316814 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b363559-7b87-4dde-a95b-8fe222208696" containerName="maas-api" Nov 03 19:22:57.317033 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.317019 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b363559-7b87-4dde-a95b-8fe222208696" containerName="maas-api" Nov 03 19:22:57.321875 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.321860 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.382346 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.374399 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-efe6c7-openshift-default-dockercfg-8mtv9\"" Nov 03 19:22:57.424766 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.424716 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg"] Nov 03 19:22:57.428483 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.428446 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-socket\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.428867 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.428504 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-data\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.428867 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.428592 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-certs\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.428867 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.428642 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrhcn\" (UniqueName: \"kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-kube-api-access-mrhcn\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.428867 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.428779 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833f3bdc-4284-4f00-8666-44d8a4a55791-istiod-ca-cert\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.429007 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.428870 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-envoy\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.429007 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.428931 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-credential-socket\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.429007 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.428980 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-podinfo\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.429111 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.429026 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-token\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.538348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.535353 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-envoy\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.538348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.535446 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-credential-socket\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.538348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.535494 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-podinfo\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.538348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.535567 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-token\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.538348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.535694 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-socket\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.538348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.535725 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-data\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.538348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.535763 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-certs\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.538348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.535792 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrhcn\" (UniqueName: \"kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-kube-api-access-mrhcn\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.538348 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.535882 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833f3bdc-4284-4f00-8666-44d8a4a55791-istiod-ca-cert\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.640426 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.637593 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-data\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.645355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.641793 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833f3bdc-4284-4f00-8666-44d8a4a55791-istiod-ca-cert\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.656424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.654612 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-credential-socket\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.656424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.654853 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-socket\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.656424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.655057 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-certs\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.659704 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.659686 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-envoy\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.660153 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.660119 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-podinfo\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.665239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.665184 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrhcn\" (UniqueName: \"kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-kube-api-access-mrhcn\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.670527 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.670498 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-token\") pod \"e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:57.941355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:57.938905 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:58.155073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:58.155026 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b363559-7b87-4dde-a95b-8fe222208696" path="/var/lib/kubelet/pods/9b363559-7b87-4dde-a95b-8fe222208696/volumes" Nov 03 19:22:58.387799 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:22:58.387762 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833f3bdc_4284_4f00_8666_44d8a4a55791.slice/crio-f1a478fe7bb1fa9d97c351ebb1d085e8dd37cb7a7469c765623fd1924ed1382d WatchSource:0}: Error finding container f1a478fe7bb1fa9d97c351ebb1d085e8dd37cb7a7469c765623fd1924ed1382d: Status 404 returned error can't find the container with id f1a478fe7bb1fa9d97c351ebb1d085e8dd37cb7a7469c765623fd1924ed1382d Nov 03 19:22:58.388889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:58.388848 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg"] Nov 03 19:22:58.391784 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:58.391738 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:22:58.391877 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:58.391830 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:22:58.391917 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:58.391874 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:22:58.452264 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:58.452236 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" event={"ID":"833f3bdc-4284-4f00-8666-44d8a4a55791","Type":"ContainerStarted","Data":"f1a478fe7bb1fa9d97c351ebb1d085e8dd37cb7a7469c765623fd1924ed1382d"} Nov 03 19:22:59.542999 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:59.542963 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" event={"ID":"833f3bdc-4284-4f00-8666-44d8a4a55791","Type":"ContainerStarted","Data":"41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82"} Nov 03 19:22:59.777371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:59.777307 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" podStartSLOduration=2.777292225 podStartE2EDuration="2.777292225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:22:59.758832611 +0000 UTC m=+2313.731393794" watchObservedRunningTime="2026-11-03 19:22:59.777292225 +0000 UTC m=+2313.749853406" Nov 03 19:22:59.942082 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:59.942050 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:22:59.947886 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:22:59.947862 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:23:00.550076 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:00.550045 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:23:00.551025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:00.551006 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:23:09.833591 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:09.833554 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj"] Nov 03 19:23:09.879847 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:09.879814 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp"] Nov 03 19:23:11.302281 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:23:11.302243 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:23:24.856155 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:24.856096 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" containerID="cri-o://390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e" gracePeriod=45 Nov 03 19:23:24.903635 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:24.903604 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" containerID="cri-o://414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222" gracePeriod=45 Nov 03 19:23:25.193276 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.193250 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:23:25.226819 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.226794 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:23:25.241492 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.241466 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-home\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-home\") pod \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " Nov 03 19:23:25.241617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.241537 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d172ec5-a832-4cba-ae9f-10b54f54b10b-kube-api-access-7pmfx\" (UniqueName: \"kubernetes.io/projected/1d172ec5-a832-4cba-ae9f-10b54f54b10b-kube-api-access-7pmfx\") pod \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " Nov 03 19:23:25.241617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.241569 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-model-cache\") pod \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " Nov 03 19:23:25.241733 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.241684 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-home\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-home\") pod \"7cc38985-372c-4466-8e01-1413d5a581d9\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " Nov 03 19:23:25.241733 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.241714 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc38985-372c-4466-8e01-1413d5a581d9-kube-api-access-lm4k7\" (UniqueName: \"kubernetes.io/projected/7cc38985-372c-4466-8e01-1413d5a581d9-kube-api-access-lm4k7\") pod \"7cc38985-372c-4466-8e01-1413d5a581d9\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " Nov 03 19:23:25.241842 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.241754 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cc38985-372c-4466-8e01-1413d5a581d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc38985-372c-4466-8e01-1413d5a581d9-tls-certs\") pod \"7cc38985-372c-4466-8e01-1413d5a581d9\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " Nov 03 19:23:25.241842 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.241750 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-home" pod "1d172ec5-a832-4cba-ae9f-10b54f54b10b" (UID: "1d172ec5-a832-4cba-ae9f-10b54f54b10b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.241948 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.241839 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-tmp-dir\") pod \"7cc38985-372c-4466-8e01-1413d5a581d9\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " Nov 03 19:23:25.241948 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.241870 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-model-cache" pod "1d172ec5-a832-4cba-ae9f-10b54f54b10b" (UID: "1d172ec5-a832-4cba-ae9f-10b54f54b10b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.242053 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.241948 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-home" pod "7cc38985-372c-4466-8e01-1413d5a581d9" (UID: "7cc38985-372c-4466-8e01-1413d5a581d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.242053 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.241980 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-dshm\") pod \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " Nov 03 19:23:25.242053 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.242013 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-model-cache\") pod \"7cc38985-372c-4466-8e01-1413d5a581d9\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " Nov 03 19:23:25.242234 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.242057 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-dshm\") pod \"7cc38985-372c-4466-8e01-1413d5a581d9\" (UID: \"7cc38985-372c-4466-8e01-1413d5a581d9\") " Nov 03 19:23:25.242234 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.242086 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tmp-dir\") pod \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " Nov 03 19:23:25.242234 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.242157 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tls-certs\") pod \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\" (UID: \"1d172ec5-a832-4cba-ae9f-10b54f54b10b\") " Nov 03 19:23:25.242475 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.242302 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-model-cache" pod "7cc38985-372c-4466-8e01-1413d5a581d9" (UID: "7cc38985-372c-4466-8e01-1413d5a581d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.242831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.242750 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-home\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.242831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.242773 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-model-cache\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.242831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.242786 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-home\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.242831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.242801 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-model-cache\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.243120 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.243026 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tmp-dir" pod "1d172ec5-a832-4cba-ae9f-10b54f54b10b" (UID: "1d172ec5-a832-4cba-ae9f-10b54f54b10b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.243442 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.243415 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-tmp-dir" pod "7cc38985-372c-4466-8e01-1413d5a581d9" (UID: "7cc38985-372c-4466-8e01-1413d5a581d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.244217 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.244190 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d172ec5-a832-4cba-ae9f-10b54f54b10b-kube-api-access-7pmfx" pod "1d172ec5-a832-4cba-ae9f-10b54f54b10b" (UID: "1d172ec5-a832-4cba-ae9f-10b54f54b10b"). InnerVolumeSpecName "kube-api-access-7pmfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:23:25.244409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.244364 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc38985-372c-4466-8e01-1413d5a581d9-kube-api-access-lm4k7" pod "7cc38985-372c-4466-8e01-1413d5a581d9" (UID: "7cc38985-372c-4466-8e01-1413d5a581d9"). InnerVolumeSpecName "kube-api-access-lm4k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:23:25.244683 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.244650 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-dshm" pod "1d172ec5-a832-4cba-ae9f-10b54f54b10b" (UID: "1d172ec5-a832-4cba-ae9f-10b54f54b10b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.244868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.244850 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tls-certs" pod "1d172ec5-a832-4cba-ae9f-10b54f54b10b" (UID: "1d172ec5-a832-4cba-ae9f-10b54f54b10b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:23:25.246037 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.246020 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc38985-372c-4466-8e01-1413d5a581d9-tls-certs" pod "7cc38985-372c-4466-8e01-1413d5a581d9" (UID: "7cc38985-372c-4466-8e01-1413d5a581d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:23:25.246316 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.246289 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-dshm" pod "7cc38985-372c-4466-8e01-1413d5a581d9" (UID: "7cc38985-372c-4466-8e01-1413d5a581d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:25.344302 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.344255 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pmfx\" (UniqueName: \"kubernetes.io/projected/1d172ec5-a832-4cba-ae9f-10b54f54b10b-kube-api-access-7pmfx\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.344302 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.344298 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm4k7\" (UniqueName: \"kubernetes.io/projected/7cc38985-372c-4466-8e01-1413d5a581d9-kube-api-access-lm4k7\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.344302 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.344314 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc38985-372c-4466-8e01-1413d5a581d9-tls-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.344565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.344350 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-tmp-dir\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.344565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.344367 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-dshm\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.344565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.344377 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc38985-372c-4466-8e01-1413d5a581d9-dshm\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.344565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.344387 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tmp-dir\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.344565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.344399 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d172ec5-a832-4cba-ae9f-10b54f54b10b-tls-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:25.803086 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.803048 5019 generic.go:356] "Generic (PLEG): container finished" podID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerID="390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e" exitCode=0 Nov 03 19:23:25.803281 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.803095 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" Nov 03 19:23:25.803281 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.803130 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" event={"ID":"1d172ec5-a832-4cba-ae9f-10b54f54b10b","Type":"ContainerDied","Data":"390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e"} Nov 03 19:23:25.803281 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.803177 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj" event={"ID":"1d172ec5-a832-4cba-ae9f-10b54f54b10b","Type":"ContainerDied","Data":"124f517931090fc208356279527aa32b5b59444f9222231b678f695650b862b7"} Nov 03 19:23:25.803281 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.803199 5019 scope.go:122] "RemoveContainer" containerID="390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e" Nov 03 19:23:25.805086 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.805064 5019 generic.go:356] "Generic (PLEG): container finished" podID="7cc38985-372c-4466-8e01-1413d5a581d9" containerID="414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222" exitCode=0 Nov 03 19:23:25.805208 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.805092 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" event={"ID":"7cc38985-372c-4466-8e01-1413d5a581d9","Type":"ContainerDied","Data":"414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222"} Nov 03 19:23:25.805208 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.805124 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" Nov 03 19:23:25.805208 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.805137 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp" event={"ID":"7cc38985-372c-4466-8e01-1413d5a581d9","Type":"ContainerDied","Data":"c3cacbe926099768d9908b07b91de1e682062e52aa7b194286d9892ed0646aa2"} Nov 03 19:23:25.816656 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.816637 5019 scope.go:122] "RemoveContainer" containerID="390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e" Nov 03 19:23:25.816949 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:23:25.816929 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e\": container with ID starting with 390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e not found: ID does not exist" containerID="390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e" Nov 03 19:23:25.817012 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.816962 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e"} err="failed to get container status \"390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e\": rpc error: code = NotFound desc = could not find container \"390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e\": container with ID starting with 390d588e2c9beee097cee2375b4a0f2cfeb8874048feb3c50ffdea1efc72185e not found: ID does not exist" Nov 03 19:23:25.817012 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.816987 5019 scope.go:122] "RemoveContainer" containerID="414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222" Nov 03 19:23:25.826231 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.826213 5019 scope.go:122] "RemoveContainer" containerID="414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222" Nov 03 19:23:25.826521 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:23:25.826500 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222\": container with ID starting with 414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222 not found: ID does not exist" containerID="414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222" Nov 03 19:23:25.826591 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.826528 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222"} err="failed to get container status \"414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222\": rpc error: code = NotFound desc = could not find container \"414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222\": container with ID starting with 414944731d07c4c1154027ebfe862c213db9f2de36e17f3cf348acf6e2632222 not found: ID does not exist" Nov 03 19:23:25.959292 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.958738 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj"] Nov 03 19:23:25.962918 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:25.962890 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/distinct-2-w4-98860e-kserve-7956b9b4db-hmdjj"] Nov 03 19:23:26.106197 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:26.106122 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp"] Nov 03 19:23:26.109372 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:26.109347 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-98860e/distinct-w4-98860e-kserve-65fddf785c-mtxwp"] Nov 03 19:23:26.151068 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:26.151040 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" path="/var/lib/kubelet/pods/1d172ec5-a832-4cba-ae9f-10b54f54b10b/volumes" Nov 03 19:23:26.151561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:26.151546 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" path="/var/lib/kubelet/pods/7cc38985-372c-4466-8e01-1413d5a581d9/volumes" Nov 03 19:23:31.391397 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.391359 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb"] Nov 03 19:23:31.391832 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.391615 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" podUID="b8ac426d-dc73-4cfa-8373-e7da9a598199" containerName="maas-api" containerID="cri-o://016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4" gracePeriod=30 Nov 03 19:23:31.744375 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.744351 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:23:31.840299 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.840270 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-metrics-certs\") pod \"b8ac426d-dc73-4cfa-8373-e7da9a598199\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " Nov 03 19:23:31.840499 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.840410 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-maas-api-tls\") pod \"b8ac426d-dc73-4cfa-8373-e7da9a598199\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " Nov 03 19:23:31.840499 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.840447 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ac426d-dc73-4cfa-8373-e7da9a598199-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ac426d-dc73-4cfa-8373-e7da9a598199-openshift-service-ca\") pod \"b8ac426d-dc73-4cfa-8373-e7da9a598199\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " Nov 03 19:23:31.840573 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.840503 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ac426d-dc73-4cfa-8373-e7da9a598199-kube-api-access-mnddr\" (UniqueName: \"kubernetes.io/projected/b8ac426d-dc73-4cfa-8373-e7da9a598199-kube-api-access-mnddr\") pod \"b8ac426d-dc73-4cfa-8373-e7da9a598199\" (UID: \"b8ac426d-dc73-4cfa-8373-e7da9a598199\") " Nov 03 19:23:31.842233 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.842199 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ac426d-dc73-4cfa-8373-e7da9a598199-openshift-service-ca" pod "b8ac426d-dc73-4cfa-8373-e7da9a598199" (UID: "b8ac426d-dc73-4cfa-8373-e7da9a598199"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:23:31.845515 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.845484 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ac426d-dc73-4cfa-8373-e7da9a598199-kube-api-access-mnddr" pod "b8ac426d-dc73-4cfa-8373-e7da9a598199" (UID: "b8ac426d-dc73-4cfa-8373-e7da9a598199"). InnerVolumeSpecName "kube-api-access-mnddr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:23:31.845733 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.845707 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-maas-api-tls" pod "b8ac426d-dc73-4cfa-8373-e7da9a598199" (UID: "b8ac426d-dc73-4cfa-8373-e7da9a598199"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:23:31.850059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.850033 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-metrics-certs" pod "b8ac426d-dc73-4cfa-8373-e7da9a598199" (UID: "b8ac426d-dc73-4cfa-8373-e7da9a598199"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:23:31.851052 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.851027 5019 generic.go:356] "Generic (PLEG): container finished" podID="b8ac426d-dc73-4cfa-8373-e7da9a598199" containerID="016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4" exitCode=0 Nov 03 19:23:31.851141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.851116 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" event={"ID":"b8ac426d-dc73-4cfa-8373-e7da9a598199","Type":"ContainerDied","Data":"016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4"} Nov 03 19:23:31.851181 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.851165 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" event={"ID":"b8ac426d-dc73-4cfa-8373-e7da9a598199","Type":"ContainerDied","Data":"4564758dc8268d837f516e03021b07fc440ee560053abab8e601aa22fcfbfc54"} Nov 03 19:23:31.851222 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.851196 5019 scope.go:122] "RemoveContainer" containerID="016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4" Nov 03 19:23:31.852430 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.852389 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb" Nov 03 19:23:31.894214 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.894187 5019 scope.go:122] "RemoveContainer" containerID="016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4" Nov 03 19:23:31.894639 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:23:31.894610 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4\": container with ID starting with 016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4 not found: ID does not exist" containerID="016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4" Nov 03 19:23:31.894753 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.894648 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4"} err="failed to get container status \"016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4\": rpc error: code = NotFound desc = could not find container \"016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4\": container with ID starting with 016c4ad156e96e73bc36e5956f51e24c4601c770acd0b3b051abbb08328b15e4 not found: ID does not exist" Nov 03 19:23:31.942822 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.942787 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:31.942822 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.942819 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ac426d-dc73-4cfa-8373-e7da9a598199-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:31.942994 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.942830 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnddr\" (UniqueName: \"kubernetes.io/projected/b8ac426d-dc73-4cfa-8373-e7da9a598199-kube-api-access-mnddr\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:31.942994 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:31.942854 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ac426d-dc73-4cfa-8373-e7da9a598199-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:32.028206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:32.028112 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb"] Nov 03 19:23:32.029453 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:32.029425 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cc3f2-74fbd5f777-lnbsb"] Nov 03 19:23:32.151805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:32.151774 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ac426d-dc73-4cfa-8373-e7da9a598199" path="/var/lib/kubelet/pods/b8ac426d-dc73-4cfa-8373-e7da9a598199/volumes" Nov 03 19:23:40.090765 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:40.090735 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt"] Nov 03 19:23:40.091284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:40.091025 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" containerID="cri-o://903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957" gracePeriod=30 Nov 03 19:23:41.298080 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:23:41.298035 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:23:45.375158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.375133 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:23:45.454843 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.454816 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-token\" (UniqueName: \"kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-token\") pod \"96db2630-ed5b-49f8-b716-53ffd18dfa33\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " Nov 03 19:23:45.455018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.454864 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-socket\") pod \"96db2630-ed5b-49f8-b716-53ffd18dfa33\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " Nov 03 19:23:45.455018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.454886 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96db2630-ed5b-49f8-b716-53ffd18dfa33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96db2630-ed5b-49f8-b716-53ffd18dfa33-istiod-ca-cert\") pod \"96db2630-ed5b-49f8-b716-53ffd18dfa33\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " Nov 03 19:23:45.455018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.454947 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-kube-api-access-dm8vc\" (UniqueName: \"kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-kube-api-access-dm8vc\") pod \"96db2630-ed5b-49f8-b716-53ffd18dfa33\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " Nov 03 19:23:45.455190 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.455048 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-credential-socket\") pod \"96db2630-ed5b-49f8-b716-53ffd18dfa33\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " Nov 03 19:23:45.455190 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.455173 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-certs\") pod \"96db2630-ed5b-49f8-b716-53ffd18dfa33\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " Nov 03 19:23:45.455294 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.455221 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-socket" pod "96db2630-ed5b-49f8-b716-53ffd18dfa33" (UID: "96db2630-ed5b-49f8-b716-53ffd18dfa33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:45.455294 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.455246 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-podinfo\") pod \"96db2630-ed5b-49f8-b716-53ffd18dfa33\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " Nov 03 19:23:45.455294 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.455274 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-envoy\") pod \"96db2630-ed5b-49f8-b716-53ffd18dfa33\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " Nov 03 19:23:45.455472 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.455365 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96db2630-ed5b-49f8-b716-53ffd18dfa33-istiod-ca-cert" pod "96db2630-ed5b-49f8-b716-53ffd18dfa33" (UID: "96db2630-ed5b-49f8-b716-53ffd18dfa33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:23:45.455472 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.455372 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-data\") pod \"96db2630-ed5b-49f8-b716-53ffd18dfa33\" (UID: \"96db2630-ed5b-49f8-b716-53ffd18dfa33\") " Nov 03 19:23:45.455574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.455474 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-certs" pod "96db2630-ed5b-49f8-b716-53ffd18dfa33" (UID: "96db2630-ed5b-49f8-b716-53ffd18dfa33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:45.455574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.455484 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-credential-socket" pod "96db2630-ed5b-49f8-b716-53ffd18dfa33" (UID: "96db2630-ed5b-49f8-b716-53ffd18dfa33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:45.455774 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.455750 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-data" pod "96db2630-ed5b-49f8-b716-53ffd18dfa33" (UID: "96db2630-ed5b-49f8-b716-53ffd18dfa33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:45.456024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.455999 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:45.456100 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.456032 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96db2630-ed5b-49f8-b716-53ffd18dfa33-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:45.456100 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.456046 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:45.456100 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.456059 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:45.456100 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.456073 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:45.457669 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.457638 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-envoy" pod "96db2630-ed5b-49f8-b716-53ffd18dfa33" (UID: "96db2630-ed5b-49f8-b716-53ffd18dfa33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:23:45.458042 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.458019 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-podinfo" pod "96db2630-ed5b-49f8-b716-53ffd18dfa33" (UID: "96db2630-ed5b-49f8-b716-53ffd18dfa33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:23:45.458139 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.458123 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-token" pod "96db2630-ed5b-49f8-b716-53ffd18dfa33" (UID: "96db2630-ed5b-49f8-b716-53ffd18dfa33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:23:45.458206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.458182 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-kube-api-access-dm8vc" pod "96db2630-ed5b-49f8-b716-53ffd18dfa33" (UID: "96db2630-ed5b-49f8-b716-53ffd18dfa33"). InnerVolumeSpecName "kube-api-access-dm8vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:23:45.557081 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.557049 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm8vc\" (UniqueName: \"kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-kube-api-access-dm8vc\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:45.557081 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.557082 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:45.557269 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.557096 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:45.557269 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.557109 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96db2630-ed5b-49f8-b716-53ffd18dfa33-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:45.590645 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.590613 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk"] Nov 03 19:23:45.959838 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.959805 5019 generic.go:356] "Generic (PLEG): container finished" podID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerID="903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957" exitCode=0 Nov 03 19:23:45.960025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.959860 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" Nov 03 19:23:45.960025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.959893 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" event={"ID":"96db2630-ed5b-49f8-b716-53ffd18dfa33","Type":"ContainerDied","Data":"903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957"} Nov 03 19:23:45.960025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.959960 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt" event={"ID":"96db2630-ed5b-49f8-b716-53ffd18dfa33","Type":"ContainerDied","Data":"99fb9f22ea4d80ca3e960d353079b97f3d700d7598d72c61599f24048fb286ae"} Nov 03 19:23:45.960025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.959992 5019 scope.go:122] "RemoveContainer" containerID="903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957" Nov 03 19:23:45.973228 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.973205 5019 scope.go:122] "RemoveContainer" containerID="e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c" Nov 03 19:23:45.985278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.985257 5019 scope.go:122] "RemoveContainer" containerID="903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957" Nov 03 19:23:45.985645 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:23:45.985625 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957\": container with ID starting with 903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957 not found: ID does not exist" containerID="903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957" Nov 03 19:23:45.985748 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.985655 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957"} err="failed to get container status \"903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957\": rpc error: code = NotFound desc = could not find container \"903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957\": container with ID starting with 903271d1ed4336d773d1cbe036f5ed0aca2a36d0ea6884bbb85d761662c7c957 not found: ID does not exist" Nov 03 19:23:45.985748 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.985678 5019 scope.go:122] "RemoveContainer" containerID="e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c" Nov 03 19:23:45.985945 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:23:45.985928 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c\": container with ID starting with e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c not found: ID does not exist" containerID="e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c" Nov 03 19:23:45.986013 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:45.985949 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c"} err="failed to get container status \"e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c\": rpc error: code = NotFound desc = could not find container \"e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c\": container with ID starting with e3d580813f7fbe1a29209586ceda5c204937c916d26a304267227db0b46c452c not found: ID does not exist" Nov 03 19:23:46.179999 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:46.177805 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt"] Nov 03 19:23:46.180316 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:46.180290 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1cc3f2-openshift-default-85794fb47f-62drt"] Nov 03 19:23:48.150883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:48.150850 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" path="/var/lib/kubelet/pods/96db2630-ed5b-49f8-b716-53ffd18dfa33/volumes" Nov 03 19:23:56.098508 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.098465 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9"] Nov 03 19:23:56.099405 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099381 5019 cpu_manager.go:414] "containerMap: removing container" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" Nov 03 19:23:56.099405 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099406 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" Nov 03 19:23:56.099625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099456 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b8ac426d-dc73-4cfa-8373-e7da9a598199" containerName="maas-api" Nov 03 19:23:56.099625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099466 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ac426d-dc73-4cfa-8373-e7da9a598199" containerName="maas-api" Nov 03 19:23:56.099625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099483 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" Nov 03 19:23:56.099625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099490 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" Nov 03 19:23:56.099625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099513 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" Nov 03 19:23:56.099625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099520 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" Nov 03 19:23:56.099925 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099662 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" Nov 03 19:23:56.099925 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099679 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ac426d-dc73-4cfa-8373-e7da9a598199" containerName="maas-api" Nov 03 19:23:56.099925 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099686 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d172ec5-a832-4cba-ae9f-10b54f54b10b" containerName="main" Nov 03 19:23:56.099925 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099694 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc38985-372c-4466-8e01-1413d5a581d9" containerName="main" Nov 03 19:23:56.099925 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.099700 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" Nov 03 19:23:56.103500 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.103480 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.168025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.167982 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9"] Nov 03 19:23:56.214700 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.214666 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c3e530-openshift-default-dockercfg-jtvxs\"" Nov 03 19:23:56.218093 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.218057 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-envoy\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.218242 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.218134 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-token\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.218377 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.218306 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddca4421-6d98-415e-9848-80ec181cb4ae-istiod-ca-cert\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.218429 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.218400 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-podinfo\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.218500 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.218464 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjwr\" (UniqueName: \"kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-kube-api-access-drjwr\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.218626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.218608 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-certs\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.218706 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.218691 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-socket\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.218754 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.218739 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-credential-socket\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.218858 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.218837 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-data\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.343367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.342304 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-token\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.343367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.342472 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddca4421-6d98-415e-9848-80ec181cb4ae-istiod-ca-cert\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.343367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.342527 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-podinfo\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.343367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.342606 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjwr\" (UniqueName: \"kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-kube-api-access-drjwr\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.343367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.342674 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-certs\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.343367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.342746 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-socket\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.343367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.342791 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-credential-socket\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.343367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.342868 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-data\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.343367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.343178 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-envoy\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.356433 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.344909 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-certs\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.356433 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.345682 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddca4421-6d98-415e-9848-80ec181cb4ae-istiod-ca-cert\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.356433 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.346535 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-socket\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.356433 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.347519 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-data\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.357665 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.357637 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-credential-socket\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.425664 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.425618 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-envoy\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.438877 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.438847 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-podinfo\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.466361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.448092 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-token\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.467572 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.467501 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjwr\" (UniqueName: \"kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-kube-api-access-drjwr\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:56.726350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:56.725828 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:57.196923 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:23:57.196889 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddca4421_6d98_415e_9848_80ec181cb4ae.slice/crio-17a2876f300d1a1a51b0ca31dbfe0f0c6b5b24898f5f83c0e5bb76c775d49dab WatchSource:0}: Error finding container 17a2876f300d1a1a51b0ca31dbfe0f0c6b5b24898f5f83c0e5bb76c775d49dab: Status 404 returned error can't find the container with id 17a2876f300d1a1a51b0ca31dbfe0f0c6b5b24898f5f83c0e5bb76c775d49dab Nov 03 19:23:57.197398 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:57.197270 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9"] Nov 03 19:23:57.200479 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:57.200441 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:23:57.200597 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:57.200526 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:23:57.200597 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:57.200571 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:23:58.050366 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:58.050304 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" event={"ID":"ddca4421-6d98-415e-9848-80ec181cb4ae","Type":"ContainerStarted","Data":"5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99"} Nov 03 19:23:58.050554 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:58.050389 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" event={"ID":"ddca4421-6d98-415e-9848-80ec181cb4ae","Type":"ContainerStarted","Data":"17a2876f300d1a1a51b0ca31dbfe0f0c6b5b24898f5f83c0e5bb76c775d49dab"} Nov 03 19:23:58.213120 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:58.213068 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" podStartSLOduration=2.213056438 podStartE2EDuration="2.213056438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:23:58.210460153 +0000 UTC m=+2372.183021336" watchObservedRunningTime="2026-11-03 19:23:58.213056438 +0000 UTC m=+2372.185617689" Nov 03 19:23:58.727275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:58.727240 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:58.733578 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:58.733551 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:59.056580 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:59.056499 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:23:59.057673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:23:59.057642 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:24:00.612564 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:00.612505 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" containerID="cri-o://cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d" gracePeriod=45 Nov 03 19:24:00.617352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:00.617306 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": EOF" Nov 03 19:24:00.909468 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:00.909440 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:24:01.029230 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.029202 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b49433-77e0-4513-bc54-c8d3d10dc443-kube-api-access-mfh5l\" (UniqueName: \"kubernetes.io/projected/33b49433-77e0-4513-bc54-c8d3d10dc443-kube-api-access-mfh5l\") pod \"33b49433-77e0-4513-bc54-c8d3d10dc443\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " Nov 03 19:24:01.029491 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.029469 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b49433-77e0-4513-bc54-c8d3d10dc443-tls-certs\" (UniqueName: \"kubernetes.io/secret/33b49433-77e0-4513-bc54-c8d3d10dc443-tls-certs\") pod \"33b49433-77e0-4513-bc54-c8d3d10dc443\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " Nov 03 19:24:01.029561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.029516 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-tmp-dir\") pod \"33b49433-77e0-4513-bc54-c8d3d10dc443\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " Nov 03 19:24:01.029618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.029551 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-dshm\") pod \"33b49433-77e0-4513-bc54-c8d3d10dc443\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " Nov 03 19:24:01.029677 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.029634 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-model-cache\") pod \"33b49433-77e0-4513-bc54-c8d3d10dc443\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " Nov 03 19:24:01.029750 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.029733 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-home\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-home\") pod \"33b49433-77e0-4513-bc54-c8d3d10dc443\" (UID: \"33b49433-77e0-4513-bc54-c8d3d10dc443\") " Nov 03 19:24:01.029899 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.029842 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-tmp-dir" pod "33b49433-77e0-4513-bc54-c8d3d10dc443" (UID: "33b49433-77e0-4513-bc54-c8d3d10dc443"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:01.030212 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.030147 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-home" pod "33b49433-77e0-4513-bc54-c8d3d10dc443" (UID: "33b49433-77e0-4513-bc54-c8d3d10dc443"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:01.030371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.030303 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-home\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.030371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.030304 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-model-cache" pod "33b49433-77e0-4513-bc54-c8d3d10dc443" (UID: "33b49433-77e0-4513-bc54-c8d3d10dc443"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:01.030533 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.030319 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-tmp-dir\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.032772 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.032744 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-dshm" pod "33b49433-77e0-4513-bc54-c8d3d10dc443" (UID: "33b49433-77e0-4513-bc54-c8d3d10dc443"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:01.033434 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.033403 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b49433-77e0-4513-bc54-c8d3d10dc443-kube-api-access-mfh5l" pod "33b49433-77e0-4513-bc54-c8d3d10dc443" (UID: "33b49433-77e0-4513-bc54-c8d3d10dc443"). InnerVolumeSpecName "kube-api-access-mfh5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:01.057549 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.054445 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b49433-77e0-4513-bc54-c8d3d10dc443-tls-certs" pod "33b49433-77e0-4513-bc54-c8d3d10dc443" (UID: "33b49433-77e0-4513-bc54-c8d3d10dc443"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:01.093381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.085354 5019 generic.go:356] "Generic (PLEG): container finished" podID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerID="cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d" exitCode=0 Nov 03 19:24:01.093381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.086212 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" Nov 03 19:24:01.093381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.089560 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" event={"ID":"33b49433-77e0-4513-bc54-c8d3d10dc443","Type":"ContainerDied","Data":"cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d"} Nov 03 19:24:01.093381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.089604 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk" event={"ID":"33b49433-77e0-4513-bc54-c8d3d10dc443","Type":"ContainerDied","Data":"c44e9aac873796f1572d2ada67aa6c27e8334691b5cc857ec4455652e4762110"} Nov 03 19:24:01.093381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.089626 5019 scope.go:122] "RemoveContainer" containerID="cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d" Nov 03 19:24:01.112592 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.110087 5019 scope.go:122] "RemoveContainer" containerID="cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d" Nov 03 19:24:01.112592 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:24:01.110444 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d\": container with ID starting with cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d not found: ID does not exist" containerID="cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d" Nov 03 19:24:01.112592 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.110481 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d"} err="failed to get container status \"cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d\": rpc error: code = NotFound desc = could not find container \"cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d\": container with ID starting with cbd322cfa99d11278af81994a4487b53c86f99cb83a35b34964501f59ba6385d not found: ID does not exist" Nov 03 19:24:01.132267 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.132157 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b49433-77e0-4513-bc54-c8d3d10dc443-tls-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.132267 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.132206 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-dshm\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.132267 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.132216 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b49433-77e0-4513-bc54-c8d3d10dc443-model-cache\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.132267 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.132228 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfh5l\" (UniqueName: \"kubernetes.io/projected/33b49433-77e0-4513-bc54-c8d3d10dc443-kube-api-access-mfh5l\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:01.246680 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.246650 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk"] Nov 03 19:24:01.251265 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:01.251230 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/distinct-w5-90a9bf-kserve-74ffbb655-sdrjk"] Nov 03 19:24:02.152532 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:02.152503 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" path="/var/lib/kubelet/pods/33b49433-77e0-4513-bc54-c8d3d10dc443/volumes" Nov 03 19:24:06.426187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:06.426105 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt"] Nov 03 19:24:06.426649 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:06.426340 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" podUID="a050632f-9924-48ed-8271-ed06f65445a6" containerName="maas-api" containerID="cri-o://e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b" gracePeriod=30 Nov 03 19:24:06.853393 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:06.840855 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:24:06.932362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:06.930426 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a050632f-9924-48ed-8271-ed06f65445a6-kube-api-access-9s5vd\" (UniqueName: \"kubernetes.io/projected/a050632f-9924-48ed-8271-ed06f65445a6-kube-api-access-9s5vd\") pod \"a050632f-9924-48ed-8271-ed06f65445a6\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " Nov 03 19:24:06.932362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:06.930632 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-maas-api-tls\") pod \"a050632f-9924-48ed-8271-ed06f65445a6\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " Nov 03 19:24:06.932362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:06.930710 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-metrics-certs\") pod \"a050632f-9924-48ed-8271-ed06f65445a6\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " Nov 03 19:24:06.932362 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:06.930761 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a050632f-9924-48ed-8271-ed06f65445a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a050632f-9924-48ed-8271-ed06f65445a6-openshift-service-ca\") pod \"a050632f-9924-48ed-8271-ed06f65445a6\" (UID: \"a050632f-9924-48ed-8271-ed06f65445a6\") " Nov 03 19:24:06.956436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:06.954474 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-maas-api-tls" pod "a050632f-9924-48ed-8271-ed06f65445a6" (UID: "a050632f-9924-48ed-8271-ed06f65445a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:06.956436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:06.954929 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a050632f-9924-48ed-8271-ed06f65445a6-openshift-service-ca" pod "a050632f-9924-48ed-8271-ed06f65445a6" (UID: "a050632f-9924-48ed-8271-ed06f65445a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:06.975520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:06.957204 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a050632f-9924-48ed-8271-ed06f65445a6-kube-api-access-9s5vd" pod "a050632f-9924-48ed-8271-ed06f65445a6" (UID: "a050632f-9924-48ed-8271-ed06f65445a6"). InnerVolumeSpecName "kube-api-access-9s5vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:06.975520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:06.962359 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-metrics-certs" pod "a050632f-9924-48ed-8271-ed06f65445a6" (UID: "a050632f-9924-48ed-8271-ed06f65445a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:07.049488 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.049390 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s5vd\" (UniqueName: \"kubernetes.io/projected/a050632f-9924-48ed-8271-ed06f65445a6-kube-api-access-9s5vd\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:07.050804 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.050783 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:07.051013 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.051001 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a050632f-9924-48ed-8271-ed06f65445a6-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:07.051114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.051103 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a050632f-9924-48ed-8271-ed06f65445a6-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:07.132890 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.132851 5019 generic.go:356] "Generic (PLEG): container finished" podID="a050632f-9924-48ed-8271-ed06f65445a6" containerID="e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b" exitCode=0 Nov 03 19:24:07.133085 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.132917 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" Nov 03 19:24:07.133085 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.132938 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" event={"ID":"a050632f-9924-48ed-8271-ed06f65445a6","Type":"ContainerDied","Data":"e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b"} Nov 03 19:24:07.133085 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.133030 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt" event={"ID":"a050632f-9924-48ed-8271-ed06f65445a6","Type":"ContainerDied","Data":"f86c06bb69135043162886a5c0f82b151096eacc9c7007eaba4032826c3d225f"} Nov 03 19:24:07.133085 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.133053 5019 scope.go:122] "RemoveContainer" containerID="e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b" Nov 03 19:24:07.144875 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.144853 5019 scope.go:122] "RemoveContainer" containerID="e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b" Nov 03 19:24:07.145175 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:24:07.145154 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b\": container with ID starting with e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b not found: ID does not exist" containerID="e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b" Nov 03 19:24:07.145241 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.145187 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b"} err="failed to get container status \"e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b\": rpc error: code = NotFound desc = could not find container \"e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b\": container with ID starting with e1cd0ed80117ca036f5e8845058feb43842033b7f49cb53e34111cd36ecf9a2b not found: ID does not exist" Nov 03 19:24:07.287520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.287476 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt"] Nov 03 19:24:07.291517 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.291491 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd1327-6666596548-tq4nt"] Nov 03 19:24:07.584527 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.584495 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv"] Nov 03 19:24:07.584978 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.584708 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" podUID="33f4064a-a444-449d-9769-fe2eb909d71f" containerName="maas-api" containerID="cri-o://6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d" gracePeriod=30 Nov 03 19:24:07.591196 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.591113 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" podUID="33f4064a-a444-449d-9769-fe2eb909d71f" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.147:8443/health\": dial tcp 10.132.0.147:8443: connect: connection refused" Nov 03 19:24:07.867871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.867847 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:24:07.967352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.966540 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f4064a-a444-449d-9769-fe2eb909d71f-kube-api-access-8crhl\" (UniqueName: \"kubernetes.io/projected/33f4064a-a444-449d-9769-fe2eb909d71f-kube-api-access-8crhl\") pod \"33f4064a-a444-449d-9769-fe2eb909d71f\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " Nov 03 19:24:07.967352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.966668 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f4064a-a444-449d-9769-fe2eb909d71f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f4064a-a444-449d-9769-fe2eb909d71f-openshift-service-ca\") pod \"33f4064a-a444-449d-9769-fe2eb909d71f\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " Nov 03 19:24:07.967352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.966760 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-metrics-certs\") pod \"33f4064a-a444-449d-9769-fe2eb909d71f\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " Nov 03 19:24:07.967352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.966849 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-maas-api-tls\") pod \"33f4064a-a444-449d-9769-fe2eb909d71f\" (UID: \"33f4064a-a444-449d-9769-fe2eb909d71f\") " Nov 03 19:24:07.984347 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.968217 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f4064a-a444-449d-9769-fe2eb909d71f-openshift-service-ca" pod "33f4064a-a444-449d-9769-fe2eb909d71f" (UID: "33f4064a-a444-449d-9769-fe2eb909d71f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:07.984347 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.976440 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-metrics-certs" pod "33f4064a-a444-449d-9769-fe2eb909d71f" (UID: "33f4064a-a444-449d-9769-fe2eb909d71f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:07.984347 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.980587 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f4064a-a444-449d-9769-fe2eb909d71f-kube-api-access-8crhl" pod "33f4064a-a444-449d-9769-fe2eb909d71f" (UID: "33f4064a-a444-449d-9769-fe2eb909d71f"). InnerVolumeSpecName "kube-api-access-8crhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:07.998622 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:07.998587 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-maas-api-tls" pod "33f4064a-a444-449d-9769-fe2eb909d71f" (UID: "33f4064a-a444-449d-9769-fe2eb909d71f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:08.068908 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.068872 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:08.068908 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.068904 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8crhl\" (UniqueName: \"kubernetes.io/projected/33f4064a-a444-449d-9769-fe2eb909d71f-kube-api-access-8crhl\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:08.069090 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.068914 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f4064a-a444-449d-9769-fe2eb909d71f-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:08.069090 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.068923 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f4064a-a444-449d-9769-fe2eb909d71f-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:08.141514 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.141435 5019 generic.go:356] "Generic (PLEG): container finished" podID="33f4064a-a444-449d-9769-fe2eb909d71f" containerID="6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d" exitCode=0 Nov 03 19:24:08.141514 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.141482 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" Nov 03 19:24:08.141700 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.141516 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" event={"ID":"33f4064a-a444-449d-9769-fe2eb909d71f","Type":"ContainerDied","Data":"6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d"} Nov 03 19:24:08.141700 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.141571 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv" event={"ID":"33f4064a-a444-449d-9769-fe2eb909d71f","Type":"ContainerDied","Data":"52902967b4e1e7a28ffbb21c6dd69f0dbc5b9350514a7643f5605b58ada8faff"} Nov 03 19:24:08.141700 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.141594 5019 scope.go:122] "RemoveContainer" containerID="6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d" Nov 03 19:24:08.155810 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.155776 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a050632f-9924-48ed-8271-ed06f65445a6" path="/var/lib/kubelet/pods/a050632f-9924-48ed-8271-ed06f65445a6/volumes" Nov 03 19:24:08.155877 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.155807 5019 scope.go:122] "RemoveContainer" containerID="6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d" Nov 03 19:24:08.156082 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:24:08.156063 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d\": container with ID starting with 6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d not found: ID does not exist" containerID="6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d" Nov 03 19:24:08.156122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.156090 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d"} err="failed to get container status \"6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d\": rpc error: code = NotFound desc = could not find container \"6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d\": container with ID starting with 6655614e573c5b598328a204babe1af934ff52b176bac5223b11910e436a409d not found: ID does not exist" Nov 03 19:24:08.297685 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.297645 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv"] Nov 03 19:24:08.323361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:08.323307 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d86d1cd0-788bc9697f-qfxpv"] Nov 03 19:24:10.152242 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:10.152206 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f4064a-a444-449d-9769-fe2eb909d71f" path="/var/lib/kubelet/pods/33f4064a-a444-449d-9769-fe2eb909d71f/volumes" Nov 03 19:24:11.297455 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:24:11.297415 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:24:14.453501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.453469 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5"] Nov 03 19:24:14.454368 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.454345 5019 cpu_manager.go:414] "containerMap: removing container" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" Nov 03 19:24:14.454368 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.454368 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96db2630-ed5b-49f8-b716-53ffd18dfa33" containerName="istio-proxy" Nov 03 19:24:14.454547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.454414 5019 cpu_manager.go:414] "containerMap: removing container" podUID="33f4064a-a444-449d-9769-fe2eb909d71f" containerName="maas-api" Nov 03 19:24:14.454547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.454423 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f4064a-a444-449d-9769-fe2eb909d71f" containerName="maas-api" Nov 03 19:24:14.454547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.454434 5019 cpu_manager.go:414] "containerMap: removing container" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" Nov 03 19:24:14.454547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.454441 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" Nov 03 19:24:14.454547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.454461 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a050632f-9924-48ed-8271-ed06f65445a6" containerName="maas-api" Nov 03 19:24:14.454547 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.454467 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a050632f-9924-48ed-8271-ed06f65445a6" containerName="maas-api" Nov 03 19:24:14.454837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.454619 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f4064a-a444-449d-9769-fe2eb909d71f" containerName="maas-api" Nov 03 19:24:14.454837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.454635 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b49433-77e0-4513-bc54-c8d3d10dc443" containerName="main" Nov 03 19:24:14.454837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.454648 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a050632f-9924-48ed-8271-ed06f65445a6" containerName="maas-api" Nov 03 19:24:14.458453 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.458432 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:14.507532 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.507506 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-efe6c7-serving-cert\"" Nov 03 19:24:14.559564 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.559524 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5"] Nov 03 19:24:14.576099 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.576067 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-maas-api-tls\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:14.576274 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.576138 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7dg\" (UniqueName: \"kubernetes.io/projected/92a817a4-2971-4470-801e-799a8f1f71e3-kube-api-access-pp7dg\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:14.576274 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.576226 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-metrics-certs\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:14.576438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.576409 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a817a4-2971-4470-801e-799a8f1f71e3-openshift-service-ca\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:14.693586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.690321 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7dg\" (UniqueName: \"kubernetes.io/projected/92a817a4-2971-4470-801e-799a8f1f71e3-kube-api-access-pp7dg\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:14.693586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.690583 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-metrics-certs\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:14.693586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.690789 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a817a4-2971-4470-801e-799a8f1f71e3-openshift-service-ca\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:14.693586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.690819 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-maas-api-tls\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:14.693586 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:24:14.691008 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-serving-cert: secret "maas-api-e2e-api-b-efe6c7-serving-cert" not found Nov 03 19:24:14.693586 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:24:14.691071 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-maas-api-tls podName:92a817a4-2971-4470-801e-799a8f1f71e3 nodeName:}" failed. No retries permitted until 2026-11-03 19:24:15.191050814 +0000 UTC m=+2389.163611993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-maas-api-tls") pod "maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" (UID: "92a817a4-2971-4470-801e-799a8f1f71e3") : secret "maas-api-e2e-api-b-efe6c7-serving-cert" not found Nov 03 19:24:14.694875 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.694849 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a817a4-2971-4470-801e-799a8f1f71e3-openshift-service-ca\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:14.705550 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.705480 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-metrics-certs\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:14.749049 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:14.749019 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7dg\" (UniqueName: \"kubernetes.io/projected/92a817a4-2971-4470-801e-799a8f1f71e3-kube-api-access-pp7dg\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:15.196885 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:15.196852 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-maas-api-tls\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:15.199605 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:15.199570 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-maas-api-tls\") pod \"maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:15.373853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:15.373817 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:15.776701 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:24:15.776673 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a817a4_2971_4470_801e_799a8f1f71e3.slice/crio-42f3f25664123ae20090f276b28234d0ed1393c81bd66c4398b6c162186ed996 WatchSource:0}: Error finding container 42f3f25664123ae20090f276b28234d0ed1393c81bd66c4398b6c162186ed996: Status 404 returned error can't find the container with id 42f3f25664123ae20090f276b28234d0ed1393c81bd66c4398b6c162186ed996 Nov 03 19:24:15.781273 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:15.781244 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5"] Nov 03 19:24:15.947829 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:15.947792 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw"] Nov 03 19:24:15.948079 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:15.948051 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" containerID="cri-o://cefe8380de958fefa16f69f5f58d95070390092e0a3e314f62a7d101c2ba1d3d" gracePeriod=30 Nov 03 19:24:16.202743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:16.202705 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" event={"ID":"92a817a4-2971-4470-801e-799a8f1f71e3","Type":"ContainerStarted","Data":"42f3f25664123ae20090f276b28234d0ed1393c81bd66c4398b6c162186ed996"} Nov 03 19:24:17.212422 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.212387 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" event={"ID":"92a817a4-2971-4470-801e-799a8f1f71e3","Type":"ContainerStarted","Data":"c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a"} Nov 03 19:24:17.212863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.212542 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:17.313123 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.313092 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv"] Nov 03 19:24:17.317568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.317547 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.360391 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.357820 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" podStartSLOduration=2.443707947 podStartE2EDuration="3.357797087s" totalImagesPullingTime="914.08914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:17.325119168 +0000 UTC m=+2391.297680350" watchObservedRunningTime="2026-11-03 19:24:17.357797087 +0000 UTC m=+2391.330358270" Nov 03 19:24:17.369356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.366268 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-socket\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.369356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.366378 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-istio-token\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.369356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.366412 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-envoy\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.369356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.366474 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501946c2-6e3d-4fb2-9123-e105014fb260-istio-podinfo\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.372359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.370837 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-certs\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.372359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.371415 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-credential-socket\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.372359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.371525 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khpvh\" (UniqueName: \"kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-kube-api-access-khpvh\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.372359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.371599 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501946c2-6e3d-4fb2-9123-e105014fb260-istiod-ca-cert\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.372359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.371704 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-data\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.515409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.509683 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-istio-token\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.515409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.509745 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-envoy\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.515409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.509815 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501946c2-6e3d-4fb2-9123-e105014fb260-istio-podinfo\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.515409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.509890 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-certs\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.515409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.509985 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-credential-socket\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.515409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.510036 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khpvh\" (UniqueName: \"kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-kube-api-access-khpvh\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.515409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.510099 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501946c2-6e3d-4fb2-9123-e105014fb260-istiod-ca-cert\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.515409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.510138 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-data\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.515409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.510262 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-socket\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.515409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.510882 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-socket\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.515409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.514581 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-credential-socket\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.522983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.517748 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-envoy\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.522983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.518760 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501946c2-6e3d-4fb2-9123-e105014fb260-istiod-ca-cert\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.522983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.519019 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-data\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.522983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.519276 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-certs\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.558379 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.557894 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501946c2-6e3d-4fb2-9123-e105014fb260-istio-podinfo\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.609347 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.602007 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-73a216-openshift-default-dockercfg-4xn9g\"" Nov 03 19:24:17.615319 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.615027 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv"] Nov 03 19:24:17.624458 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.623934 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khpvh\" (UniqueName: \"kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-kube-api-access-khpvh\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.631712 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.631689 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-istio-token\") pod \"e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:17.632063 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:17.632042 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:18.062721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:18.062693 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv"] Nov 03 19:24:18.065285 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:24:18.065235 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501946c2_6e3d_4fb2_9123_e105014fb260.slice/crio-b7cd2bb022c63354d0fa7b8ec7b276bd0fab11d63614f7a80defe62107a0fd52 WatchSource:0}: Error finding container b7cd2bb022c63354d0fa7b8ec7b276bd0fab11d63614f7a80defe62107a0fd52: Status 404 returned error can't find the container with id b7cd2bb022c63354d0fa7b8ec7b276bd0fab11d63614f7a80defe62107a0fd52 Nov 03 19:24:18.068052 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:18.068013 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:24:18.068168 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:18.068080 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:24:18.068168 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:18.068109 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:24:18.221054 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:18.221017 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" event={"ID":"501946c2-6e3d-4fb2-9123-e105014fb260","Type":"ContainerStarted","Data":"c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64"} Nov 03 19:24:18.221568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:18.221084 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" event={"ID":"501946c2-6e3d-4fb2-9123-e105014fb260","Type":"ContainerStarted","Data":"b7cd2bb022c63354d0fa7b8ec7b276bd0fab11d63614f7a80defe62107a0fd52"} Nov 03 19:24:18.334442 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:18.334301 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" podStartSLOduration=1.334287509 podStartE2EDuration="1.334287509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:18.333705047 +0000 UTC m=+2392.306266226" watchObservedRunningTime="2026-11-03 19:24:18.334287509 +0000 UTC m=+2392.306848690" Nov 03 19:24:18.632459 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:18.632380 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:19.402496 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:19.402464 5019 patch_prober.go:29] interesting pod/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:24:19.402974 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:19.402534 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:24:19.639653 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:19.639622 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:20.235183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:20.235150 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:20.236259 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:20.236237 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:24:21.344296 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.343535 5019 generic.go:356] "Generic (PLEG): container finished" podID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerID="cefe8380de958fefa16f69f5f58d95070390092e0a3e314f62a7d101c2ba1d3d" exitCode=0 Nov 03 19:24:21.344794 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.344661 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" event={"ID":"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996","Type":"ContainerDied","Data":"cefe8380de958fefa16f69f5f58d95070390092e0a3e314f62a7d101c2ba1d3d"} Nov 03 19:24:21.344794 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.344721 5019 scope.go:122] "RemoveContainer" containerID="77319ec8bfb0a30b0c793c586cae4beea50580b908a35be3482660502ada0ec1" Nov 03 19:24:21.345356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.345257 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:24:21.527943 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.527898 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-socket\") pod \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " Nov 03 19:24:21.528147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.528083 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-certs\") pod \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " Nov 03 19:24:21.528147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.528139 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-podinfo\") pod \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " Nov 03 19:24:21.528279 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.528176 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-data\") pod \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " Nov 03 19:24:21.528279 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.528232 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-credential-socket\") pod \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " Nov 03 19:24:21.528405 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.528344 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-envoy\") pod \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " Nov 03 19:24:21.528467 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.528440 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-kube-api-access-l2gvs\" (UniqueName: \"kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-kube-api-access-l2gvs\") pod \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " Nov 03 19:24:21.528574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.528557 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istiod-ca-cert\") pod \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " Nov 03 19:24:21.528641 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.528617 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-token\" (UniqueName: \"kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-token\") pod \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\" (UID: \"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996\") " Nov 03 19:24:21.530157 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.530124 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-socket" pod "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" (UID: "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:21.530424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.530386 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-certs" pod "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" (UID: "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:21.536922 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.536893 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-credential-socket" pod "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" (UID: "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:21.537102 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.537057 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-data" pod "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" (UID: "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:21.537579 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.537553 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istiod-ca-cert" pod "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" (UID: "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:21.540395 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.540358 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-token" pod "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" (UID: "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:21.541728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.541698 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-podinfo" pod "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" (UID: "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:24:21.541913 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.541887 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-envoy" pod "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" (UID: "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:21.542130 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.542105 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-kube-api-access-l2gvs" pod "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" (UID: "b99b4137-3f69-4ca3-b3c2-e3c3a33f6996"). InnerVolumeSpecName "kube-api-access-l2gvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:21.631646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.631557 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:21.631646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.631593 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:21.631646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.631609 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:21.631646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.631624 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:21.631891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.631650 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:21.631891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.631664 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:21.631891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.631677 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2gvs\" (UniqueName: \"kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-kube-api-access-l2gvs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:21.631891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.631694 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:21.631891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:21.631707 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:22.352224 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:22.352195 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" Nov 03 19:24:22.352224 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:22.352208 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw" event={"ID":"b99b4137-3f69-4ca3-b3c2-e3c3a33f6996","Type":"ContainerDied","Data":"af15a50ef867bd20f2d41c44a8e35616fdf5e73feed457cf61cc125d00c9119e"} Nov 03 19:24:22.352805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:22.352267 5019 scope.go:122] "RemoveContainer" containerID="cefe8380de958fefa16f69f5f58d95070390092e0a3e314f62a7d101c2ba1d3d" Nov 03 19:24:22.626518 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:22.626446 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw"] Nov 03 19:24:22.633082 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:22.633045 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dd1327-openshift-default-7745c7fb65-wflmw"] Nov 03 19:24:23.226989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:23.226958 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:24:24.153314 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:24.153280 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" path="/var/lib/kubelet/pods/b99b4137-3f69-4ca3-b3c2-e3c3a33f6996/volumes" Nov 03 19:24:29.773967 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:29.773937 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:24:29.778301 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:29.778277 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:24:41.297594 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:24:41.297553 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:24:52.555515 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:52.555481 5019 generic.go:356] "Generic (PLEG): container finished" podID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerID="41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82" exitCode=0 Nov 03 19:24:52.555950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:52.555544 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" event={"ID":"833f3bdc-4284-4f00-8666-44d8a4a55791","Type":"ContainerDied","Data":"41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82"} Nov 03 19:24:52.556007 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:52.555988 5019 scope.go:122] "RemoveContainer" containerID="41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82" Nov 03 19:24:52.557848 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:52.557816 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:24:52.557937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:52.557890 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:24:52.557937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:52.557928 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:24:53.588650 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:53.588608 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" event={"ID":"833f3bdc-4284-4f00-8666-44d8a4a55791","Type":"ContainerStarted","Data":"06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007"} Nov 03 19:24:53.942574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:53.942545 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:24:53.942811 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:53.942787 5019 patch_prober.go:29] interesting pod/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 19:24:53.942897 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:53.942851 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 19:24:54.942691 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:54.942656 5019 patch_prober.go:29] interesting pod/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 19:24:54.943081 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:54.942716 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 19:24:55.949358 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:55.948374 5019 patch_prober.go:29] interesting pod/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 19:24:55.949358 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:55.948479 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 19:24:56.942203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:56.942172 5019 patch_prober.go:29] interesting pod/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 19:24:56.942393 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:56.942219 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 19:24:57.942863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:57.942827 5019 patch_prober.go:29] interesting pod/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 19:24:57.943262 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:57.942883 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 19:24:58.947035 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:58.947004 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:24:58.947446 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:58.947295 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:24:58.948104 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:24:58.948087 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:25:06.369833 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.369772 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6"] Nov 03 19:25:06.370513 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.370497 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" Nov 03 19:25:06.370513 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.370514 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" Nov 03 19:25:06.370513 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.370526 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" Nov 03 19:25:06.370513 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.370531 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" Nov 03 19:25:06.370754 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.370667 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" Nov 03 19:25:06.370754 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.370682 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99b4137-3f69-4ca3-b3c2-e3c3a33f6996" containerName="istio-proxy" Nov 03 19:25:06.375554 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.375535 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:06.424588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.424557 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c3e530-serving-cert\"" Nov 03 19:25:06.430898 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.430860 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrrpd\" (UniqueName: \"kubernetes.io/projected/b54643ca-250e-49db-b008-f7f6e15aca04-kube-api-access-mrrpd\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:06.430994 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.430915 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-metrics-certs\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:06.431149 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.431127 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-maas-api-tls\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:06.431192 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.431158 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54643ca-250e-49db-b008-f7f6e15aca04-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:06.474677 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.474644 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6"] Nov 03 19:25:06.532663 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.532633 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-maas-api-tls\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:06.532663 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.532665 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54643ca-250e-49db-b008-f7f6e15aca04-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:06.532918 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.532745 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrrpd\" (UniqueName: \"kubernetes.io/projected/b54643ca-250e-49db-b008-f7f6e15aca04-kube-api-access-mrrpd\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:06.532918 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.532790 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-metrics-certs\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:06.532918 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:25:06.532801 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-serving-cert: secret "maas-api-e2e-inf-b-c3e530-serving-cert" not found Nov 03 19:25:06.532918 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:25:06.532863 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-maas-api-tls podName:b54643ca-250e-49db-b008-f7f6e15aca04 nodeName:}" failed. No retries permitted until 2026-11-03 19:25:07.032843307 +0000 UTC m=+2441.005404505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-maas-api-tls") pod "maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" (UID: "b54643ca-250e-49db-b008-f7f6e15aca04") : secret "maas-api-e2e-inf-b-c3e530-serving-cert" not found Nov 03 19:25:06.533524 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.533495 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54643ca-250e-49db-b008-f7f6e15aca04-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:06.535568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.535542 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-metrics-certs\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:06.591039 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:06.591003 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrrpd\" (UniqueName: \"kubernetes.io/projected/b54643ca-250e-49db-b008-f7f6e15aca04-kube-api-access-mrrpd\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:07.038657 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:07.038620 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-maas-api-tls\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:07.041141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:07.041112 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-maas-api-tls\") pod \"maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:07.290109 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:07.290031 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:07.666824 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:07.666688 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6"] Nov 03 19:25:07.668427 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:25:07.668403 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54643ca_250e_49db_b008_f7f6e15aca04.slice/crio-eec6f509649a6ccb2573348335f0e642fed1a77ca71e71e3c1ce31e23505a820 WatchSource:0}: Error finding container eec6f509649a6ccb2573348335f0e642fed1a77ca71e71e3c1ce31e23505a820: Status 404 returned error can't find the container with id eec6f509649a6ccb2573348335f0e642fed1a77ca71e71e3c1ce31e23505a820 Nov 03 19:25:07.729247 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:07.729213 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" event={"ID":"b54643ca-250e-49db-b008-f7f6e15aca04","Type":"ContainerStarted","Data":"eec6f509649a6ccb2573348335f0e642fed1a77ca71e71e3c1ce31e23505a820"} Nov 03 19:25:08.737394 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:08.737355 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" event={"ID":"b54643ca-250e-49db-b008-f7f6e15aca04","Type":"ContainerStarted","Data":"cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa"} Nov 03 19:25:08.737829 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:08.737585 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:08.852020 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:08.851968 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" podStartSLOduration=1.903626934 podStartE2EDuration="2.851953127s" totalImagesPullingTime="948.326193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:25:08.85068441 +0000 UTC m=+2442.823245591" watchObservedRunningTime="2026-11-03 19:25:08.851953127 +0000 UTC m=+2442.824514308" Nov 03 19:25:11.300471 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:25:11.300434 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:25:14.750087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:14.750060 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:25:22.462691 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:22.462654 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np"] Nov 03 19:25:22.463240 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:22.462912 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" containerID="cri-o://64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99" gracePeriod=30 Nov 03 19:25:24.901373 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:24.901335 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4"] Nov 03 19:25:24.905951 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:24.905926 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:24.954732 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:24.954698 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-73a216-serving-cert\"" Nov 03 19:25:25.004428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.004395 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4"] Nov 03 19:25:25.020539 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.020496 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-maas-api-tls\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.020733 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.020561 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-metrics-certs\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.020733 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.020662 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vj7k\" (UniqueName: \"kubernetes.io/projected/81073a4c-136f-485e-91ec-329f36057350-kube-api-access-6vj7k\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.020829 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.020799 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81073a4c-136f-485e-91ec-329f36057350-openshift-service-ca\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.122446 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.122410 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81073a4c-136f-485e-91ec-329f36057350-openshift-service-ca\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.122611 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.122536 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-maas-api-tls\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.122611 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.122571 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-metrics-certs\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.122710 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.122617 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vj7k\" (UniqueName: \"kubernetes.io/projected/81073a4c-136f-485e-91ec-329f36057350-kube-api-access-6vj7k\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.122710 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:25:25.122691 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-serving-cert: secret "maas-api-e2e-inf-a-73a216-serving-cert" not found Nov 03 19:25:25.122814 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:25:25.122766 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-maas-api-tls podName:81073a4c-136f-485e-91ec-329f36057350 nodeName:}" failed. No retries permitted until 2026-11-03 19:25:25.622745041 +0000 UTC m=+2459.595306222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-maas-api-tls") pod "maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" (UID: "81073a4c-136f-485e-91ec-329f36057350") : secret "maas-api-e2e-inf-a-73a216-serving-cert" not found Nov 03 19:25:25.123100 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.123075 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81073a4c-136f-485e-91ec-329f36057350-openshift-service-ca\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.125438 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.125407 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-metrics-certs\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.188917 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.188881 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vj7k\" (UniqueName: \"kubernetes.io/projected/81073a4c-136f-485e-91ec-329f36057350-kube-api-access-6vj7k\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.629846 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.629759 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-maas-api-tls\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.632310 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.632283 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-maas-api-tls\") pod \"maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:25.694004 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.693970 5019 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:25:25.694135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.694034 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:25:25.830351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:25.823803 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:26.214939 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:26.214907 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4"] Nov 03 19:25:26.217079 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:25:26.217043 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81073a4c_136f_485e_91ec_329f36057350.slice/crio-03d106113b3931d3a94a5b0414f8f415ebd7682614b3316268364dc181582444 WatchSource:0}: Error finding container 03d106113b3931d3a94a5b0414f8f415ebd7682614b3316268364dc181582444: Status 404 returned error can't find the container with id 03d106113b3931d3a94a5b0414f8f415ebd7682614b3316268364dc181582444 Nov 03 19:25:26.859147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:26.859109 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" event={"ID":"81073a4c-136f-485e-91ec-329f36057350","Type":"ContainerStarted","Data":"03d106113b3931d3a94a5b0414f8f415ebd7682614b3316268364dc181582444"} Nov 03 19:25:27.756704 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.756672 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:25:27.864135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.864041 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-socket\") pod \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " Nov 03 19:25:27.864135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.864091 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-token\" (UniqueName: \"kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-token\") pod \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " Nov 03 19:25:27.864135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.864122 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istiod-ca-cert\") pod \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " Nov 03 19:25:27.864458 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.864251 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-credential-socket\") pod \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " Nov 03 19:25:27.864458 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.864379 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-envoy\") pod \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " Nov 03 19:25:27.864458 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.864388 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-socket" pod "39acdc92-a6a2-488c-b2c2-6a6133f0ce38" (UID: "39acdc92-a6a2-488c-b2c2-6a6133f0ce38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:27.864458 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.864411 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-certs\") pod \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " Nov 03 19:25:27.864458 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.864443 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-data\") pod \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " Nov 03 19:25:27.864714 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.864561 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-podinfo\") pod \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " Nov 03 19:25:27.864714 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.864597 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-kube-api-access-t7ds4\" (UniqueName: \"kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-kube-api-access-t7ds4\") pod \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\" (UID: \"39acdc92-a6a2-488c-b2c2-6a6133f0ce38\") " Nov 03 19:25:27.864714 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.864665 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istiod-ca-cert" pod "39acdc92-a6a2-488c-b2c2-6a6133f0ce38" (UID: "39acdc92-a6a2-488c-b2c2-6a6133f0ce38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:27.866051 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.864930 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-data" pod "39acdc92-a6a2-488c-b2c2-6a6133f0ce38" (UID: "39acdc92-a6a2-488c-b2c2-6a6133f0ce38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:27.866051 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.865175 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:27.866051 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.865196 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:27.866051 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.865210 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:27.866051 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.865492 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-certs" pod "39acdc92-a6a2-488c-b2c2-6a6133f0ce38" (UID: "39acdc92-a6a2-488c-b2c2-6a6133f0ce38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:27.866051 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.866019 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-credential-socket" pod "39acdc92-a6a2-488c-b2c2-6a6133f0ce38" (UID: "39acdc92-a6a2-488c-b2c2-6a6133f0ce38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:27.867050 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.866980 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-envoy" pod "39acdc92-a6a2-488c-b2c2-6a6133f0ce38" (UID: "39acdc92-a6a2-488c-b2c2-6a6133f0ce38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:27.867183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.867089 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-token" pod "39acdc92-a6a2-488c-b2c2-6a6133f0ce38" (UID: "39acdc92-a6a2-488c-b2c2-6a6133f0ce38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:27.867483 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.867462 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-kube-api-access-t7ds4" pod "39acdc92-a6a2-488c-b2c2-6a6133f0ce38" (UID: "39acdc92-a6a2-488c-b2c2-6a6133f0ce38"). InnerVolumeSpecName "kube-api-access-t7ds4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:27.867799 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.867776 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-podinfo" pod "39acdc92-a6a2-488c-b2c2-6a6133f0ce38" (UID: "39acdc92-a6a2-488c-b2c2-6a6133f0ce38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:25:27.874187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.873931 5019 generic.go:356] "Generic (PLEG): container finished" podID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerID="64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99" exitCode=0 Nov 03 19:25:27.874187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.873987 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" Nov 03 19:25:27.874187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.873983 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" event={"ID":"39acdc92-a6a2-488c-b2c2-6a6133f0ce38","Type":"ContainerDied","Data":"64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99"} Nov 03 19:25:27.874187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.874043 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np" event={"ID":"39acdc92-a6a2-488c-b2c2-6a6133f0ce38","Type":"ContainerDied","Data":"68026dcf4c673c4e95da6fb1b0e17c8207e1de6fcc3fe1ace1091daf364592b0"} Nov 03 19:25:27.874187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.874074 5019 scope.go:122] "RemoveContainer" containerID="64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99" Nov 03 19:25:27.876477 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.876454 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" event={"ID":"81073a4c-136f-485e-91ec-329f36057350","Type":"ContainerStarted","Data":"e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e"} Nov 03 19:25:27.876632 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.876599 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:27.895125 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.895100 5019 scope.go:122] "RemoveContainer" containerID="3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94" Nov 03 19:25:27.906472 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.906420 5019 scope.go:122] "RemoveContainer" containerID="64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99" Nov 03 19:25:27.906808 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:25:27.906787 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99\": container with ID starting with 64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99 not found: ID does not exist" containerID="64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99" Nov 03 19:25:27.906907 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.906818 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99"} err="failed to get container status \"64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99\": rpc error: code = NotFound desc = could not find container \"64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99\": container with ID starting with 64d97158ecf2471b2248a15e4537711cd6576ac682e8931d99e347043519fd99 not found: ID does not exist" Nov 03 19:25:27.906907 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.906838 5019 scope.go:122] "RemoveContainer" containerID="3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94" Nov 03 19:25:27.907133 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:25:27.907114 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94\": container with ID starting with 3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94 not found: ID does not exist" containerID="3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94" Nov 03 19:25:27.907189 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.907138 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94"} err="failed to get container status \"3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94\": rpc error: code = NotFound desc = could not find container \"3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94\": container with ID starting with 3f07e8b8e01d4066438a15484055bd9d0db649c0a2a221c1891cc5052bd62f94 not found: ID does not exist" Nov 03 19:25:27.967142 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.967109 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:27.967142 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.967143 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:27.967402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.967160 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:27.967402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.967174 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:27.967402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.967190 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:27.967402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:27.967205 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7ds4\" (UniqueName: \"kubernetes.io/projected/39acdc92-a6a2-488c-b2c2-6a6133f0ce38-kube-api-access-t7ds4\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:28.027589 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:28.027555 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np"] Nov 03 19:25:28.030845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:28.030821 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-489np"] Nov 03 19:25:28.137999 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:28.137883 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" podStartSLOduration=3.116389205 podStartE2EDuration="4.137868905s" totalImagesPullingTime="1.0214797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:25:28.136168226 +0000 UTC m=+2462.108729409" watchObservedRunningTime="2026-11-03 19:25:28.137868905 +0000 UTC m=+2462.110430065" Nov 03 19:25:28.152761 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:28.152720 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" path="/var/lib/kubelet/pods/39acdc92-a6a2-488c-b2c2-6a6133f0ce38/volumes" Nov 03 19:25:33.892368 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:33.892317 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:25:40.883402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:40.881960 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2"] Nov 03 19:25:40.883402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:40.882189 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" podUID="4410c3a0-9ebe-4f0b-88e8-45f2625284c7" containerName="maas-api" containerID="cri-o://19939e781b5ced82fa6558d02f6bf1664c7c788b61bf83ebe9d071c0bc9a2fdc" gracePeriod=30 Nov 03 19:25:41.034236 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.034201 5019 generic.go:356] "Generic (PLEG): container finished" podID="4410c3a0-9ebe-4f0b-88e8-45f2625284c7" containerID="19939e781b5ced82fa6558d02f6bf1664c7c788b61bf83ebe9d071c0bc9a2fdc" exitCode=0 Nov 03 19:25:41.034427 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.034242 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" event={"ID":"4410c3a0-9ebe-4f0b-88e8-45f2625284c7","Type":"ContainerDied","Data":"19939e781b5ced82fa6558d02f6bf1664c7c788b61bf83ebe9d071c0bc9a2fdc"} Nov 03 19:25:41.307320 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.307291 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 19:25:41.327988 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:25:41.327959 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:25:41.464350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.461804 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-openshift-service-ca\") pod \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " Nov 03 19:25:41.464350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.461999 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-metrics-certs\") pod \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " Nov 03 19:25:41.464350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.462120 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-kube-api-access-gg555\" (UniqueName: \"kubernetes.io/projected/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-kube-api-access-gg555\") pod \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " Nov 03 19:25:41.464350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.462234 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-maas-api-tls\") pod \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\" (UID: \"4410c3a0-9ebe-4f0b-88e8-45f2625284c7\") " Nov 03 19:25:41.464708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.464664 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-openshift-service-ca" pod "4410c3a0-9ebe-4f0b-88e8-45f2625284c7" (UID: "4410c3a0-9ebe-4f0b-88e8-45f2625284c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:41.491179 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.491150 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-maas-api-tls" pod "4410c3a0-9ebe-4f0b-88e8-45f2625284c7" (UID: "4410c3a0-9ebe-4f0b-88e8-45f2625284c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:25:41.492896 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.492814 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-kube-api-access-gg555" pod "4410c3a0-9ebe-4f0b-88e8-45f2625284c7" (UID: "4410c3a0-9ebe-4f0b-88e8-45f2625284c7"). InnerVolumeSpecName "kube-api-access-gg555". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:41.533797 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.521094 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-metrics-certs" pod "4410c3a0-9ebe-4f0b-88e8-45f2625284c7" (UID: "4410c3a0-9ebe-4f0b-88e8-45f2625284c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:25:41.604350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.603690 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:41.604350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.603728 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg555\" (UniqueName: \"kubernetes.io/projected/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-kube-api-access-gg555\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:41.604350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.603747 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:41.604350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:41.603760 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4410c3a0-9ebe-4f0b-88e8-45f2625284c7-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:42.082349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:42.080151 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" event={"ID":"4410c3a0-9ebe-4f0b-88e8-45f2625284c7","Type":"ContainerDied","Data":"460cac34826daba73fb645d7787ab67ffa41648670d8808da10edce983049b9e"} Nov 03 19:25:42.082349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:42.080230 5019 scope.go:122] "RemoveContainer" containerID="19939e781b5ced82fa6558d02f6bf1664c7c788b61bf83ebe9d071c0bc9a2fdc" Nov 03 19:25:42.082349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:42.080432 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2" Nov 03 19:25:42.298877 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:42.298835 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2"] Nov 03 19:25:42.343083 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:42.343006 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-98860e-6d8b4d6cd4-ph8v2"] Nov 03 19:25:44.151937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:44.151902 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4410c3a0-9ebe-4f0b-88e8-45f2625284c7" path="/var/lib/kubelet/pods/4410c3a0-9ebe-4f0b-88e8-45f2625284c7/volumes" Nov 03 19:25:45.098469 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:45.098434 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:25:45.098686 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:45.098492 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:25:47.122180 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:47.122142 5019 generic.go:356] "Generic (PLEG): container finished" podID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerID="5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99" exitCode=0 Nov 03 19:25:47.122644 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:47.122216 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" event={"ID":"ddca4421-6d98-415e-9848-80ec181cb4ae","Type":"ContainerDied","Data":"5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99"} Nov 03 19:25:47.122704 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:47.122656 5019 scope.go:122] "RemoveContainer" containerID="5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99" Nov 03 19:25:47.124312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:47.124283 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:25:47.124439 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:47.124368 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:25:47.124439 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:47.124405 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:25:48.251086 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:48.249826 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" event={"ID":"ddca4421-6d98-415e-9848-80ec181cb4ae","Type":"ContainerStarted","Data":"f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0"} Nov 03 19:25:48.726608 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:48.726577 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:25:48.726786 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:48.726742 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 19:25:48.726837 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:48.726787 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 19:25:48.801537 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:48.801505 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq"] Nov 03 19:25:48.801736 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:48.801717 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" containerName="istio-proxy" containerID="cri-o://94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4" gracePeriod=30 Nov 03 19:25:49.727485 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:49.727449 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 19:25:49.727886 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:49.727504 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 19:25:50.137204 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:50.137128 5019 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:25:50.137204 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:50.137177 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:25:50.726976 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:50.726936 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 19:25:50.727144 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:50.727004 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 19:25:51.037618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:51.037533 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9"] Nov 03 19:25:51.038067 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:51.037764 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" containerName="istio-proxy" containerID="cri-o://c6e79d61c3fad0bd462c60647b1eacee049486c589b829dd0327732917ee3174" gracePeriod=30 Nov 03 19:25:51.041244 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:51.041219 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj"] Nov 03 19:25:51.042024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:51.041993 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerName="istio-proxy" containerID="cri-o://e4a44decea15ea87466514bcba77bb5fd29896f0c599872b0a8de44e321e5d73" gracePeriod=30 Nov 03 19:25:51.179352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:51.178391 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:25:51.179352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:51.178469 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:25:51.727140 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:51.727096 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 19:25:51.727369 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:51.727177 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 19:25:52.739593 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:52.737836 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 19:25:52.739593 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:52.737900 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 19:25:53.378698 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.378661 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26"] Nov 03 19:25:53.379663 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.379633 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4410c3a0-9ebe-4f0b-88e8-45f2625284c7" containerName="maas-api" Nov 03 19:25:53.379663 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.379657 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4410c3a0-9ebe-4f0b-88e8-45f2625284c7" containerName="maas-api" Nov 03 19:25:53.379871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.379689 5019 cpu_manager.go:414] "containerMap: removing container" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" Nov 03 19:25:53.379871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.379697 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" Nov 03 19:25:53.379871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.379708 5019 cpu_manager.go:414] "containerMap: removing container" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" Nov 03 19:25:53.379871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.379716 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" Nov 03 19:25:53.380104 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.379904 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4410c3a0-9ebe-4f0b-88e8-45f2625284c7" containerName="maas-api" Nov 03 19:25:53.380104 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.379918 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" Nov 03 19:25:53.380104 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.379929 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="39acdc92-a6a2-488c-b2c2-6a6133f0ce38" containerName="istio-proxy" Nov 03 19:25:53.384285 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.384263 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.444723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.444688 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3ba842-d764-4169-90fc-364512d8ebca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.444930 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.444833 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.445008 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.444937 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3ba842-d764-4169-90fc-364512d8ebca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.445066 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.445005 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.445066 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.445031 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.445155 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.445129 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.445223 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.445194 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.445287 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.445253 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzljp\" (UniqueName: \"kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-kube-api-access-kzljp\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.445383 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.445288 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.546510 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.546473 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzljp\" (UniqueName: \"kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-kube-api-access-kzljp\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.546676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.546520 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.546676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.546611 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3ba842-d764-4169-90fc-364512d8ebca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.546676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.546655 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.546831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.546688 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3ba842-d764-4169-90fc-364512d8ebca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.546831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.546723 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.546831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.546748 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.546831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.546808 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.547055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.546886 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.547132 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.547111 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.547198 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.547153 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.547259 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.547203 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.547322 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.547305 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.547471 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.547452 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3ba842-d764-4169-90fc-364512d8ebca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.549129 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.549103 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.549299 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.549276 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3ba842-d764-4169-90fc-364512d8ebca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.589528 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.589499 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26"] Nov 03 19:25:53.606372 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.605885 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzljp\" (UniqueName: \"kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-kube-api-access-kzljp\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.606372 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.606313 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-6zg26\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.698912 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.698885 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:53.798920 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.798888 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:25:53.800169 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.800014 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:25:53.801152 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:53.801136 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:25:54.218401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.218375 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:25:54.226773 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.226750 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26"] Nov 03 19:25:54.227093 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:25:54.227070 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3ba842_d764_4169_90fc_364512d8ebca.slice/crio-774ae79b46d4f2958d26673b9352da0cb1278ed3670585ecd94f753a351784b5 WatchSource:0}: Error finding container 774ae79b46d4f2958d26673b9352da0cb1278ed3670585ecd94f753a351784b5: Status 404 returned error can't find the container with id 774ae79b46d4f2958d26673b9352da0cb1278ed3670585ecd94f753a351784b5 Nov 03 19:25:54.229543 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.229507 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:25:54.229628 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.229571 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:25:54.229628 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.229602 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:25:54.271596 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.271571 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-data\") pod \"c83d496d-1fb9-4860-897c-b312023a1b47\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " Nov 03 19:25:54.271720 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.271643 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-socket\") pod \"c83d496d-1fb9-4860-897c-b312023a1b47\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " Nov 03 19:25:54.271720 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.271702 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-kube-api-access-fqhjt\" (UniqueName: \"kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-kube-api-access-fqhjt\") pod \"c83d496d-1fb9-4860-897c-b312023a1b47\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " Nov 03 19:25:54.271913 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.271768 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c83d496d-1fb9-4860-897c-b312023a1b47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c83d496d-1fb9-4860-897c-b312023a1b47-istio-podinfo\") pod \"c83d496d-1fb9-4860-897c-b312023a1b47\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " Nov 03 19:25:54.271913 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.271801 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-envoy\") pod \"c83d496d-1fb9-4860-897c-b312023a1b47\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " Nov 03 19:25:54.271913 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.271852 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-data" pod "c83d496d-1fb9-4860-897c-b312023a1b47" (UID: "c83d496d-1fb9-4860-897c-b312023a1b47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:54.271913 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.271859 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-certs\") pod \"c83d496d-1fb9-4860-897c-b312023a1b47\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " Nov 03 19:25:54.271913 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.271895 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c83d496d-1fb9-4860-897c-b312023a1b47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c83d496d-1fb9-4860-897c-b312023a1b47-istiod-ca-cert\") pod \"c83d496d-1fb9-4860-897c-b312023a1b47\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " Nov 03 19:25:54.272366 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.271920 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-istio-token\" (UniqueName: \"kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-istio-token\") pod \"c83d496d-1fb9-4860-897c-b312023a1b47\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " Nov 03 19:25:54.272366 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.272069 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-credential-socket\") pod \"c83d496d-1fb9-4860-897c-b312023a1b47\" (UID: \"c83d496d-1fb9-4860-897c-b312023a1b47\") " Nov 03 19:25:54.272366 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.272171 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-certs" pod "c83d496d-1fb9-4860-897c-b312023a1b47" (UID: "c83d496d-1fb9-4860-897c-b312023a1b47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:54.272591 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.272572 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:54.272655 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.272593 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:54.272780 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.272762 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-credential-socket" pod "c83d496d-1fb9-4860-897c-b312023a1b47" (UID: "c83d496d-1fb9-4860-897c-b312023a1b47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:54.273073 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.273041 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83d496d-1fb9-4860-897c-b312023a1b47-istiod-ca-cert" pod "c83d496d-1fb9-4860-897c-b312023a1b47" (UID: "c83d496d-1fb9-4860-897c-b312023a1b47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:54.275153 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.275131 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-socket" pod "c83d496d-1fb9-4860-897c-b312023a1b47" (UID: "c83d496d-1fb9-4860-897c-b312023a1b47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:54.275437 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.275381 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-istio-token" pod "c83d496d-1fb9-4860-897c-b312023a1b47" (UID: "c83d496d-1fb9-4860-897c-b312023a1b47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:54.275541 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.275526 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-kube-api-access-fqhjt" pod "c83d496d-1fb9-4860-897c-b312023a1b47" (UID: "c83d496d-1fb9-4860-897c-b312023a1b47"). InnerVolumeSpecName "kube-api-access-fqhjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:54.275609 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.275587 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-envoy" pod "c83d496d-1fb9-4860-897c-b312023a1b47" (UID: "c83d496d-1fb9-4860-897c-b312023a1b47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:54.276124 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.276095 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c83d496d-1fb9-4860-897c-b312023a1b47-istio-podinfo" pod "c83d496d-1fb9-4860-897c-b312023a1b47" (UID: "c83d496d-1fb9-4860-897c-b312023a1b47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:25:54.317787 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.317761 5019 generic.go:356] "Generic (PLEG): container finished" podID="c83d496d-1fb9-4860-897c-b312023a1b47" containerID="94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4" exitCode=0 Nov 03 19:25:54.317906 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.317816 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" Nov 03 19:25:54.317906 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.317851 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" event={"ID":"c83d496d-1fb9-4860-897c-b312023a1b47","Type":"ContainerDied","Data":"94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4"} Nov 03 19:25:54.318020 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.317920 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq" event={"ID":"c83d496d-1fb9-4860-897c-b312023a1b47","Type":"ContainerDied","Data":"d8ec4ad3e950af94e63c9b0e78ba760dc412eaa2262a16510810ff0245d457bc"} Nov 03 19:25:54.318020 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.317952 5019 scope.go:122] "RemoveContainer" containerID="94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4" Nov 03 19:25:54.320439 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.320410 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" event={"ID":"ad3ba842-d764-4169-90fc-364512d8ebca","Type":"ContainerStarted","Data":"eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1"} Nov 03 19:25:54.324439 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.324406 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" event={"ID":"ad3ba842-d764-4169-90fc-364512d8ebca","Type":"ContainerStarted","Data":"774ae79b46d4f2958d26673b9352da0cb1278ed3670585ecd94f753a351784b5"} Nov 03 19:25:54.329252 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.329233 5019 scope.go:122] "RemoveContainer" containerID="94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4" Nov 03 19:25:54.329635 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:25:54.329607 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4\": container with ID starting with 94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4 not found: ID does not exist" containerID="94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4" Nov 03 19:25:54.329740 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.329638 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4"} err="failed to get container status \"94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4\": rpc error: code = NotFound desc = could not find container \"94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4\": container with ID starting with 94429fe828d339918705a9948a453016e4f87ba849026a1361c20518db7a93c4 not found: ID does not exist" Nov 03 19:25:54.374285 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.374007 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c83d496d-1fb9-4860-897c-b312023a1b47-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:54.374285 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.374042 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:54.374285 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.374059 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:54.374285 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.374074 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:54.374285 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.374088 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqhjt\" (UniqueName: \"kubernetes.io/projected/c83d496d-1fb9-4860-897c-b312023a1b47-kube-api-access-fqhjt\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:54.374285 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.374115 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c83d496d-1fb9-4860-897c-b312023a1b47-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:54.374285 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.374131 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c83d496d-1fb9-4860-897c-b312023a1b47-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:54.445413 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.445365 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" podStartSLOduration=1.445350779 podStartE2EDuration="1.445350779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:25:54.443550654 +0000 UTC m=+2488.416111835" watchObservedRunningTime="2026-11-03 19:25:54.445350779 +0000 UTC m=+2488.417911958" Nov 03 19:25:54.550097 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.550001 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq"] Nov 03 19:25:54.555814 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.555788 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-ntwrq"] Nov 03 19:25:54.699563 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:54.699530 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:25:55.702453 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:55.702413 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6zg26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:25:55.702965 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:55.702536 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:25:56.155757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.155724 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" path="/var/lib/kubelet/pods/c83d496d-1fb9-4860-897c-b312023a1b47/volumes" Nov 03 19:25:56.341250 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.340744 5019 generic.go:356] "Generic (PLEG): container finished" podID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerID="e4a44decea15ea87466514bcba77bb5fd29896f0c599872b0a8de44e321e5d73" exitCode=0 Nov 03 19:25:56.341250 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.340869 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" event={"ID":"c5791d78-831c-4fd7-9d30-1aaa69049ae0","Type":"ContainerDied","Data":"e4a44decea15ea87466514bcba77bb5fd29896f0c599872b0a8de44e321e5d73"} Nov 03 19:25:56.342888 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.342857 5019 generic.go:356] "Generic (PLEG): container finished" podID="b903d93f-7b45-486c-8879-7a9843c324ae" containerID="c6e79d61c3fad0bd462c60647b1eacee049486c589b829dd0327732917ee3174" exitCode=0 Nov 03 19:25:56.343993 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.343959 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" event={"ID":"b903d93f-7b45-486c-8879-7a9843c324ae","Type":"ContainerDied","Data":"c6e79d61c3fad0bd462c60647b1eacee049486c589b829dd0327732917ee3174"} Nov 03 19:25:56.426440 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.426416 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:25:56.455170 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.455145 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:25:56.510781 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.510753 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-envoy\") pod \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " Nov 03 19:25:56.510948 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.510845 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-certs\") pod \"b903d93f-7b45-486c-8879-7a9843c324ae\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " Nov 03 19:25:56.510948 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.510872 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-socket\") pod \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " Nov 03 19:25:56.510948 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.510898 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-data\") pod \"b903d93f-7b45-486c-8879-7a9843c324ae\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " Nov 03 19:25:56.510948 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.510940 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-envoy\") pod \"b903d93f-7b45-486c-8879-7a9843c324ae\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " Nov 03 19:25:56.511179 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.510995 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-socket\") pod \"b903d93f-7b45-486c-8879-7a9843c324ae\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " Nov 03 19:25:56.511179 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511032 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istiod-ca-cert\") pod \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " Nov 03 19:25:56.511179 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511088 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-kube-api-access-7mm49\" (UniqueName: \"kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-kube-api-access-7mm49\") pod \"b903d93f-7b45-486c-8879-7a9843c324ae\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " Nov 03 19:25:56.511179 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511143 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-credential-socket\") pod \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " Nov 03 19:25:56.511179 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511153 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-data" pod "b903d93f-7b45-486c-8879-7a9843c324ae" (UID: "b903d93f-7b45-486c-8879-7a9843c324ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:56.511179 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511165 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-socket" pod "c5791d78-831c-4fd7-9d30-1aaa69049ae0" (UID: "c5791d78-831c-4fd7-9d30-1aaa69049ae0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:56.511179 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511176 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-certs" pod "b903d93f-7b45-486c-8879-7a9843c324ae" (UID: "b903d93f-7b45-486c-8879-7a9843c324ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:56.511587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511176 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-certs\") pod \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " Nov 03 19:25:56.511587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511361 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-istio-token\" (UniqueName: \"kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-istio-token\") pod \"b903d93f-7b45-486c-8879-7a9843c324ae\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " Nov 03 19:25:56.511587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511416 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-podinfo\") pod \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " Nov 03 19:25:56.511587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511436 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-certs" pod "c5791d78-831c-4fd7-9d30-1aaa69049ae0" (UID: "c5791d78-831c-4fd7-9d30-1aaa69049ae0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:56.511587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511450 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-credential-socket" pod "c5791d78-831c-4fd7-9d30-1aaa69049ae0" (UID: "c5791d78-831c-4fd7-9d30-1aaa69049ae0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:56.511587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511463 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-data\") pod \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " Nov 03 19:25:56.511587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511465 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-socket" pod "b903d93f-7b45-486c-8879-7a9843c324ae" (UID: "b903d93f-7b45-486c-8879-7a9843c324ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:56.511587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511514 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b903d93f-7b45-486c-8879-7a9843c324ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b903d93f-7b45-486c-8879-7a9843c324ae-istiod-ca-cert\") pod \"b903d93f-7b45-486c-8879-7a9843c324ae\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " Nov 03 19:25:56.511587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511546 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-credential-socket\") pod \"b903d93f-7b45-486c-8879-7a9843c324ae\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " Nov 03 19:25:56.511587 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511583 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b903d93f-7b45-486c-8879-7a9843c324ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b903d93f-7b45-486c-8879-7a9843c324ae-istio-podinfo\") pod \"b903d93f-7b45-486c-8879-7a9843c324ae\" (UID: \"b903d93f-7b45-486c-8879-7a9843c324ae\") " Nov 03 19:25:56.512117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511608 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istiod-ca-cert" pod "c5791d78-831c-4fd7-9d30-1aaa69049ae0" (UID: "c5791d78-831c-4fd7-9d30-1aaa69049ae0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:56.512117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511688 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-data" pod "c5791d78-831c-4fd7-9d30-1aaa69049ae0" (UID: "c5791d78-831c-4fd7-9d30-1aaa69049ae0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:56.512117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511613 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-kube-api-access-btt5w\" (UniqueName: \"kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-kube-api-access-btt5w\") pod \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " Nov 03 19:25:56.512117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511773 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-token\" (UniqueName: \"kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-token\") pod \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\" (UID: \"c5791d78-831c-4fd7-9d30-1aaa69049ae0\") " Nov 03 19:25:56.512117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.511889 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-credential-socket" pod "b903d93f-7b45-486c-8879-7a9843c324ae" (UID: "b903d93f-7b45-486c-8879-7a9843c324ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:56.512418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.512142 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b903d93f-7b45-486c-8879-7a9843c324ae-istiod-ca-cert" pod "b903d93f-7b45-486c-8879-7a9843c324ae" (UID: "b903d93f-7b45-486c-8879-7a9843c324ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:56.512418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.512289 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.512418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.512308 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.512418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.512371 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.512418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.512387 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b903d93f-7b45-486c-8879-7a9843c324ae-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.512418 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.512405 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.512792 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.512425 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.512792 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.512441 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.512792 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.512455 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.512792 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.512468 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.512792 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.512482 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.513697 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.513672 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-envoy" pod "c5791d78-831c-4fd7-9d30-1aaa69049ae0" (UID: "c5791d78-831c-4fd7-9d30-1aaa69049ae0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:56.514029 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.514011 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-kube-api-access-7mm49" pod "b903d93f-7b45-486c-8879-7a9843c324ae" (UID: "b903d93f-7b45-486c-8879-7a9843c324ae"). InnerVolumeSpecName "kube-api-access-7mm49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:56.514278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.514256 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-podinfo" pod "c5791d78-831c-4fd7-9d30-1aaa69049ae0" (UID: "c5791d78-831c-4fd7-9d30-1aaa69049ae0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:25:56.514409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.514285 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-istio-token" pod "b903d93f-7b45-486c-8879-7a9843c324ae" (UID: "b903d93f-7b45-486c-8879-7a9843c324ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:56.514470 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.514449 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-token" pod "c5791d78-831c-4fd7-9d30-1aaa69049ae0" (UID: "c5791d78-831c-4fd7-9d30-1aaa69049ae0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:56.514572 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.514549 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-kube-api-access-btt5w" pod "c5791d78-831c-4fd7-9d30-1aaa69049ae0" (UID: "c5791d78-831c-4fd7-9d30-1aaa69049ae0"). InnerVolumeSpecName "kube-api-access-btt5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:56.515092 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.515066 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-envoy" pod "b903d93f-7b45-486c-8879-7a9843c324ae" (UID: "b903d93f-7b45-486c-8879-7a9843c324ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:56.515177 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.515110 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b903d93f-7b45-486c-8879-7a9843c324ae-istio-podinfo" pod "b903d93f-7b45-486c-8879-7a9843c324ae" (UID: "b903d93f-7b45-486c-8879-7a9843c324ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:25:56.613964 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.613872 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.613964 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.613912 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.613964 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.613928 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b903d93f-7b45-486c-8879-7a9843c324ae-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.613964 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.613942 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btt5w\" (UniqueName: \"kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-kube-api-access-btt5w\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.613964 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.613959 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.614261 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.613972 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5791d78-831c-4fd7-9d30-1aaa69049ae0-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.614261 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.613985 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b903d93f-7b45-486c-8879-7a9843c324ae-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.614261 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.613999 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mm49\" (UniqueName: \"kubernetes.io/projected/b903d93f-7b45-486c-8879-7a9843c324ae-kube-api-access-7mm49\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:56.699280 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.699239 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6zg26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:25:56.699534 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:56.699321 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:25:57.350826 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:57.350798 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" Nov 03 19:25:57.350826 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:57.350820 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9" event={"ID":"b903d93f-7b45-486c-8879-7a9843c324ae","Type":"ContainerDied","Data":"3080063c9627af94d76953eb7de6c531e71e968e918c9f7316e28dc8888719f9"} Nov 03 19:25:57.351392 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:57.350880 5019 scope.go:122] "RemoveContainer" containerID="c6e79d61c3fad0bd462c60647b1eacee049486c589b829dd0327732917ee3174" Nov 03 19:25:57.352512 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:57.352490 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" event={"ID":"c5791d78-831c-4fd7-9d30-1aaa69049ae0","Type":"ContainerDied","Data":"9698c8555b86274da6a5b67c2ee40d6420518039e32c685aba42d9eb990aa6e5"} Nov 03 19:25:57.352614 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:57.352575 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj" Nov 03 19:25:57.385799 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:57.382845 5019 scope.go:122] "RemoveContainer" containerID="e4a44decea15ea87466514bcba77bb5fd29896f0c599872b0a8de44e321e5d73" Nov 03 19:25:57.528523 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:57.528482 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9"] Nov 03 19:25:57.533475 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:57.533103 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-gbrn9"] Nov 03 19:25:57.631817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:57.631782 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj"] Nov 03 19:25:57.636108 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:57.636038 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-zsvvj"] Nov 03 19:25:57.699745 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:57.699707 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6zg26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:25:57.699907 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:57.699763 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:25:58.152018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:58.151974 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" path="/var/lib/kubelet/pods/b903d93f-7b45-486c-8879-7a9843c324ae/volumes" Nov 03 19:25:58.152630 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:58.152596 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" path="/var/lib/kubelet/pods/c5791d78-831c-4fd7-9d30-1aaa69049ae0/volumes" Nov 03 19:25:58.699585 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:58.699553 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6zg26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:25:58.700025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:58.699605 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:25:59.699575 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:59.699538 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6zg26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:25:59.699785 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:25:59.699616 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:26:00.700197 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:00.700161 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6zg26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:26:00.700968 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:00.700224 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:26:01.700009 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:01.699973 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6zg26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:26:01.700265 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:01.700037 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:26:02.594468 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.594421 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm"] Nov 03 19:26:02.595682 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.595656 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" containerName="istio-proxy" Nov 03 19:26:02.595682 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.595681 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" containerName="istio-proxy" Nov 03 19:26:02.595921 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.595699 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerName="istio-proxy" Nov 03 19:26:02.595921 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.595707 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerName="istio-proxy" Nov 03 19:26:02.595921 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.595729 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" containerName="istio-proxy" Nov 03 19:26:02.595921 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.595759 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" containerName="istio-proxy" Nov 03 19:26:02.596161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.595947 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83d496d-1fb9-4860-897c-b312023a1b47" containerName="istio-proxy" Nov 03 19:26:02.596161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.595977 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b903d93f-7b45-486c-8879-7a9843c324ae" containerName="istio-proxy" Nov 03 19:26:02.596161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.595988 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5791d78-831c-4fd7-9d30-1aaa69049ae0" containerName="istio-proxy" Nov 03 19:26:02.601339 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.601300 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.650640 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.650601 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-332dac-openshift-default-dockercfg-w2jm5\"" Nov 03 19:26:02.701939 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.701908 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm"] Nov 03 19:26:02.727517 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.727477 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-podinfo\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.727675 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.727555 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-socket\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.727675 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.727596 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-data\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.727675 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.727669 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istiod-ca-cert\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.727803 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.727696 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-certs\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.727803 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.727731 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-envoy\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.727803 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.727786 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc6cs\" (UniqueName: \"kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-kube-api-access-dc6cs\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.727901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.727821 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-credential-socket\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.727901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.727891 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-token\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.829622 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.829581 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc6cs\" (UniqueName: \"kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-kube-api-access-dc6cs\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.829801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.829631 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-credential-socket\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.829801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.829674 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-token\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.829801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.829744 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-podinfo\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.829801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.829795 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-socket\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.829990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.829834 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-data\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.829990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.829912 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istiod-ca-cert\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.829990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.829937 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-certs\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.829990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.829965 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-envoy\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.830313 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.830280 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-certs\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.830473 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.830319 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-socket\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.830473 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.830410 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-data\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.830613 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.830512 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-credential-socket\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.830723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.830703 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istiod-ca-cert\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.832524 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.832492 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-podinfo\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.832719 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.832697 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-envoy\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.869450 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.869374 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:26:02.871084 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.871063 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:26:02.872433 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.872412 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:26:02.887703 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.887675 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-token\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.887839 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.887682 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc6cs\" (UniqueName: \"kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-kube-api-access-dc6cs\") pod \"e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:02.918676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:02.918654 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:03.348586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:03.348554 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm"] Nov 03 19:26:03.349512 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:26:03.349472 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d500e8_f34b_4fe8_ad68_363552ebd6f6.slice/crio-4adbcacd5abb64d239616d51e405790cc2a8f884eca91029b8d387dd01a09729 WatchSource:0}: Error finding container 4adbcacd5abb64d239616d51e405790cc2a8f884eca91029b8d387dd01a09729: Status 404 returned error can't find the container with id 4adbcacd5abb64d239616d51e405790cc2a8f884eca91029b8d387dd01a09729 Nov 03 19:26:03.352152 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:03.352116 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:26:03.352270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:03.352202 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:26:03.352270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:03.352249 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:26:03.425866 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:03.425839 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" event={"ID":"11d500e8-f34b-4fe8-ad68-363552ebd6f6","Type":"ContainerStarted","Data":"4adbcacd5abb64d239616d51e405790cc2a8f884eca91029b8d387dd01a09729"} Nov 03 19:26:04.456352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.444738 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" event={"ID":"11d500e8-f34b-4fe8-ad68-363552ebd6f6","Type":"ContainerStarted","Data":"447f5a726744a69bc2837d6f00d08164115bdb24a3fa0ad4dd2f0803c1ac348b"} Nov 03 19:26:04.584355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.584209 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" podStartSLOduration=2.584188017 podStartE2EDuration="2.584188017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:04.557472195 +0000 UTC m=+2498.530033377" watchObservedRunningTime="2026-11-03 19:26:04.584188017 +0000 UTC m=+2498.556749200" Nov 03 19:26:04.657569 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.657533 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8"] Nov 03 19:26:04.681671 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.681636 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.722711 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.722637 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8"] Nov 03 19:26:04.731646 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.731618 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1410ed-openshift-default-dockercfg-df98m\"" Nov 03 19:26:04.784352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.761089 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istiod-ca-cert\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.784880 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.784849 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnps7\" (UniqueName: \"kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-kube-api-access-bnps7\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.788514 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.788474 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-socket\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.801081 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.801029 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-data\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.801443 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.801424 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-token\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.801749 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.801730 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-credential-socket\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.801918 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.801902 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-certs\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.802078 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.802063 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-podinfo\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.802223 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.802208 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-envoy\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.904345 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.904295 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnps7\" (UniqueName: \"kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-kube-api-access-bnps7\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.904550 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.904358 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-socket\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.904550 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.904407 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-data\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.904550 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.904483 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-token\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.904718 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.904570 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-credential-socket\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.904718 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.904602 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-certs\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.904718 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.904625 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-podinfo\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.904718 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.904661 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-envoy\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.904939 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.904763 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istiod-ca-cert\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.904939 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.904811 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-data\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.905128 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.905010 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-certs\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.905199 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.905178 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-credential-socket\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.905439 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.905408 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-socket\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.905643 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.905614 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istiod-ca-cert\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.907417 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.907393 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-envoy\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.907639 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.907619 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-podinfo\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.919500 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.919476 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:04.921278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.921251 5019 patch_prober.go:29] interesting pod/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:26:04.921397 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.921306 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" podUID="11d500e8-f34b-4fe8-ad68-363552ebd6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:26:04.994444 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.976060 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-token\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:04.998435 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:04.998399 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnps7\" (UniqueName: \"kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-kube-api-access-bnps7\") pod \"e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:05.004898 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:05.004869 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:05.459240 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:05.459209 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8"] Nov 03 19:26:05.498361 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:26:05.498302 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea234ecc_29a0_4bdf_9784_44d3ea7f25e3.slice/crio-b5ef35f915a6b6d17adf1219cba4c488e10055a3f021434aed14f72df1793e1e WatchSource:0}: Error finding container b5ef35f915a6b6d17adf1219cba4c488e10055a3f021434aed14f72df1793e1e: Status 404 returned error can't find the container with id b5ef35f915a6b6d17adf1219cba4c488e10055a3f021434aed14f72df1793e1e Nov 03 19:26:05.501188 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:05.501154 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:26:05.501290 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:05.501241 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:26:05.501290 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:05.501275 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:26:05.920230 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:05.920082 5019 patch_prober.go:29] interesting pod/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:26:05.920230 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:05.920144 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" podUID="11d500e8-f34b-4fe8-ad68-363552ebd6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:26:06.458238 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:06.458197 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" event={"ID":"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3","Type":"ContainerStarted","Data":"5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d"} Nov 03 19:26:06.458482 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:06.458265 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" event={"ID":"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3","Type":"ContainerStarted","Data":"b5ef35f915a6b6d17adf1219cba4c488e10055a3f021434aed14f72df1793e1e"} Nov 03 19:26:06.584349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:06.584161 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" podStartSLOduration=2.584141953 podStartE2EDuration="2.584141953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:06.582817691 +0000 UTC m=+2500.555378875" watchObservedRunningTime="2026-11-03 19:26:06.584141953 +0000 UTC m=+2500.556703135" Nov 03 19:26:06.926352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:06.922788 5019 patch_prober.go:29] interesting pod/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:26:06.926352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:06.922868 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" podUID="11d500e8-f34b-4fe8-ad68-363552ebd6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:26:07.006187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:07.005682 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:07.010375 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:07.008936 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:26:07.010375 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:07.009006 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" podUID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:26:07.925352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:07.924384 5019 patch_prober.go:29] interesting pod/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:26:07.925352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:07.924466 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" podUID="11d500e8-f34b-4fe8-ad68-363552ebd6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:26:08.012350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:08.006699 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:26:08.012350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:08.006770 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" podUID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:26:08.924353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:08.921821 5019 patch_prober.go:29] interesting pod/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:26:08.924353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:08.921881 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" podUID="11d500e8-f34b-4fe8-ad68-363552ebd6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:26:09.006361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:09.005756 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:26:09.006361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:09.005821 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" podUID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:26:09.920119 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:09.920075 5019 patch_prober.go:29] interesting pod/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:26:09.920313 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:09.920155 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" podUID="11d500e8-f34b-4fe8-ad68-363552ebd6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:26:10.006300 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:10.006269 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:26:10.006543 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:10.006520 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" podUID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:26:10.924276 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:10.924235 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:10.924626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:10.924598 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:10.925508 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:10.925487 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:26:11.006168 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:11.006136 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:26:11.006310 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:11.006192 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" podUID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:26:11.330153 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:26:11.330070 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:26:12.007894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:12.007853 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:26:12.008412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:12.008321 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" podUID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:26:13.010831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:13.010796 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:13.011321 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:13.011295 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:13.012035 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:13.012019 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:26:21.276493 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:21.276450 5019 patch_prober.go:29] interesting pod/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:26:21.276992 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:21.276534 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:26:22.647727 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:22.647698 5019 generic.go:356] "Generic (PLEG): container finished" podID="501946c2-6e3d-4fb2-9123-e105014fb260" containerID="c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64" exitCode=0 Nov 03 19:26:22.648102 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:22.647742 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" event={"ID":"501946c2-6e3d-4fb2-9123-e105014fb260","Type":"ContainerDied","Data":"c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64"} Nov 03 19:26:22.648146 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:22.648132 5019 scope.go:122] "RemoveContainer" containerID="c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64" Nov 03 19:26:22.649943 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:22.649903 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:26:22.650041 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:22.649993 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:26:22.650080 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:22.650038 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:26:23.656731 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:23.656698 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" event={"ID":"501946c2-6e3d-4fb2-9123-e105014fb260","Type":"ContainerStarted","Data":"ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc"} Nov 03 19:26:24.633054 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:24.633022 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:26:24.633278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:24.633150 5019 patch_prober.go:29] interesting pod/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 19:26:24.633278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:24.633203 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 19:26:25.633504 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:25.633471 5019 patch_prober.go:29] interesting pod/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 19:26:25.634006 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:25.633532 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 19:26:26.633248 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:26.633210 5019 patch_prober.go:29] interesting pod/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 19:26:26.633500 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:26.633264 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 19:26:27.632645 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:27.632612 5019 patch_prober.go:29] interesting pod/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 19:26:27.633041 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:27.632679 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 19:26:28.633154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:28.633110 5019 patch_prober.go:29] interesting pod/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 19:26:28.633611 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:28.633187 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 19:26:29.637193 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:29.637163 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:26:29.637673 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:29.637567 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:26:29.638117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:29.638098 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:26:41.298182 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:26:41.298139 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:26:45.427785 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.427752 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt"] Nov 03 19:26:45.434439 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.434418 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.528807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.528772 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-22ed3b-openshift-default-dockercfg-4vw92\"" Nov 03 19:26:45.542890 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.542839 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt"] Nov 03 19:26:45.562025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.561308 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtl9d\" (UniqueName: \"kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-kube-api-access-qtl9d\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.562025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.561450 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-credential-socket\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.562025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.561507 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-certs\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.562025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.561584 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-socket\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.562025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.561621 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istiod-ca-cert\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.562025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.561677 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-data\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.562025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.561716 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-envoy\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.562025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.561740 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-podinfo\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.562025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.561827 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-token\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.666826 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.666221 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtl9d\" (UniqueName: \"kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-kube-api-access-qtl9d\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.666826 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.666823 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-credential-socket\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.677546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.667789 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-credential-socket\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.677546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.667890 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-certs\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.677546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.668070 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-socket\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.677546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.668115 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istiod-ca-cert\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.677546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.668174 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-data\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.677546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.668214 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-envoy\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.677546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.668238 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-podinfo\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.677546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.669029 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istiod-ca-cert\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.677546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.669202 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-token\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.677546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.673651 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-socket\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.677546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.674217 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-data\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.677546 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.674945 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-certs\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.684926 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.684866 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-podinfo\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.686306 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.686268 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-envoy\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.723947 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.723918 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtl9d\" (UniqueName: \"kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-kube-api-access-qtl9d\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.734492 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.734465 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-token\") pod \"e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:45.750602 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:45.750578 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:46.172159 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:46.172131 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt"] Nov 03 19:26:46.172634 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:26:46.172609 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0629a8f5_b876_46b6_bb0f_5d73ae38a489.slice/crio-1a549b59b0b94639515a48b4a1c099a08935a3c9b08bd61eb0b0d288d43239d5 WatchSource:0}: Error finding container 1a549b59b0b94639515a48b4a1c099a08935a3c9b08bd61eb0b0d288d43239d5: Status 404 returned error can't find the container with id 1a549b59b0b94639515a48b4a1c099a08935a3c9b08bd61eb0b0d288d43239d5 Nov 03 19:26:46.175185 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:46.175153 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:26:46.175275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:46.175217 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:26:46.175275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:46.175247 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:26:46.855968 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:46.855932 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" event={"ID":"0629a8f5-b876-46b6-bb0f-5d73ae38a489","Type":"ContainerStarted","Data":"7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93"} Nov 03 19:26:46.856391 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:46.856024 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" event={"ID":"0629a8f5-b876-46b6-bb0f-5d73ae38a489","Type":"ContainerStarted","Data":"1a549b59b0b94639515a48b4a1c099a08935a3c9b08bd61eb0b0d288d43239d5"} Nov 03 19:26:47.018625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:47.018579 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" podStartSLOduration=2.018563855 podStartE2EDuration="2.018563855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:47.014859441 +0000 UTC m=+2540.987420624" watchObservedRunningTime="2026-11-03 19:26:47.018563855 +0000 UTC m=+2540.991125039" Nov 03 19:26:47.751406 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:47.751373 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:47.757708 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:47.757684 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:47.862983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:47.862947 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:26:47.863908 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:26:47.863889 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:27:11.298561 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:27:11.298461 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:27:26.243831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.243793 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9"] Nov 03 19:27:26.244428 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.243999 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" podUID="e494384c-192a-4c1e-8bd8-5c0389817cc5" containerName="maas-api" containerID="cri-o://974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971" gracePeriod=30 Nov 03 19:27:26.560469 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.560440 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:27:26.632215 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.632174 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e494384c-192a-4c1e-8bd8-5c0389817cc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e494384c-192a-4c1e-8bd8-5c0389817cc5-openshift-service-ca\") pod \"e494384c-192a-4c1e-8bd8-5c0389817cc5\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " Nov 03 19:27:26.633004 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.632863 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e494384c-192a-4c1e-8bd8-5c0389817cc5-openshift-service-ca" pod "e494384c-192a-4c1e-8bd8-5c0389817cc5" (UID: "e494384c-192a-4c1e-8bd8-5c0389817cc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:26.633212 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.633194 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-maas-api-tls\") pod \"e494384c-192a-4c1e-8bd8-5c0389817cc5\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " Nov 03 19:27:26.633823 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.633804 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e494384c-192a-4c1e-8bd8-5c0389817cc5-kube-api-access-r8r49\" (UniqueName: \"kubernetes.io/projected/e494384c-192a-4c1e-8bd8-5c0389817cc5-kube-api-access-r8r49\") pod \"e494384c-192a-4c1e-8bd8-5c0389817cc5\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " Nov 03 19:27:26.634314 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.634297 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-metrics-certs\") pod \"e494384c-192a-4c1e-8bd8-5c0389817cc5\" (UID: \"e494384c-192a-4c1e-8bd8-5c0389817cc5\") " Nov 03 19:27:26.635387 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.635364 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e494384c-192a-4c1e-8bd8-5c0389817cc5-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:26.703908 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.702400 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-maas-api-tls" pod "e494384c-192a-4c1e-8bd8-5c0389817cc5" (UID: "e494384c-192a-4c1e-8bd8-5c0389817cc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:26.703908 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.702704 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-metrics-certs" pod "e494384c-192a-4c1e-8bd8-5c0389817cc5" (UID: "e494384c-192a-4c1e-8bd8-5c0389817cc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:26.703908 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.702813 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e494384c-192a-4c1e-8bd8-5c0389817cc5-kube-api-access-r8r49" pod "e494384c-192a-4c1e-8bd8-5c0389817cc5" (UID: "e494384c-192a-4c1e-8bd8-5c0389817cc5"). InnerVolumeSpecName "kube-api-access-r8r49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:26.741145 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.741110 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:26.741145 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.741147 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8r49\" (UniqueName: \"kubernetes.io/projected/e494384c-192a-4c1e-8bd8-5c0389817cc5-kube-api-access-r8r49\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:26.741403 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:26.741163 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e494384c-192a-4c1e-8bd8-5c0389817cc5-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:27.188399 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:27.188366 5019 generic.go:356] "Generic (PLEG): container finished" podID="e494384c-192a-4c1e-8bd8-5c0389817cc5" containerID="974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971" exitCode=0 Nov 03 19:27:27.188622 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:27.188448 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" Nov 03 19:27:27.188622 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:27.188488 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" event={"ID":"e494384c-192a-4c1e-8bd8-5c0389817cc5","Type":"ContainerDied","Data":"974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971"} Nov 03 19:27:27.188622 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:27.188545 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9" event={"ID":"e494384c-192a-4c1e-8bd8-5c0389817cc5","Type":"ContainerDied","Data":"91834e212c7ff97fa1b363941318b8d6f54df81d994dcb4d4bdf5dc4642f80a6"} Nov 03 19:27:27.188622 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:27.188581 5019 scope.go:122] "RemoveContainer" containerID="974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971" Nov 03 19:27:27.201247 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:27.201223 5019 scope.go:122] "RemoveContainer" containerID="974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971" Nov 03 19:27:27.201645 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:27:27.201624 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971\": container with ID starting with 974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971 not found: ID does not exist" containerID="974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971" Nov 03 19:27:27.201730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:27.201653 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971"} err="failed to get container status \"974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971\": rpc error: code = NotFound desc = could not find container \"974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971\": container with ID starting with 974e2ee256d01115dbabf999a266a9b883979f82cc7b44afc114367425eee971 not found: ID does not exist" Nov 03 19:27:27.366451 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:27.363079 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9"] Nov 03 19:27:27.406425 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:27.406391 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2cbe3-5854d79855-fv5g9"] Nov 03 19:27:28.160999 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:28.160956 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e494384c-192a-4c1e-8bd8-5c0389817cc5" path="/var/lib/kubelet/pods/e494384c-192a-4c1e-8bd8-5c0389817cc5/volumes" Nov 03 19:27:41.305435 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:27:41.305390 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:27:51.487928 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.487890 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2"] Nov 03 19:27:51.488891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.488868 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e494384c-192a-4c1e-8bd8-5c0389817cc5" containerName="maas-api" Nov 03 19:27:51.488891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.488891 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e494384c-192a-4c1e-8bd8-5c0389817cc5" containerName="maas-api" Nov 03 19:27:51.489108 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.489095 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e494384c-192a-4c1e-8bd8-5c0389817cc5" containerName="maas-api" Nov 03 19:27:51.492636 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.492616 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:51.541696 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.541673 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-22ed3b-serving-cert\"" Nov 03 19:27:51.593284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.593249 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2"] Nov 03 19:27:51.619413 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.619372 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xltv\" (UniqueName: \"kubernetes.io/projected/3dbb737f-ccec-4959-af11-33486428f42f-kube-api-access-5xltv\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:51.619590 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.619447 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-metrics-certs\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:51.619590 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.619534 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbb737f-ccec-4959-af11-33486428f42f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:51.619721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.619657 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-maas-api-tls\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:51.720617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.720580 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xltv\" (UniqueName: \"kubernetes.io/projected/3dbb737f-ccec-4959-af11-33486428f42f-kube-api-access-5xltv\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:51.720780 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.720645 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-metrics-certs\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:51.720780 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.720721 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbb737f-ccec-4959-af11-33486428f42f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:51.720905 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.720826 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-maas-api-tls\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:51.721007 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:27:51.720992 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-serving-cert: secret "maas-api-e2e-inf-b-22ed3b-serving-cert" not found Nov 03 19:27:51.721089 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:27:51.721074 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-maas-api-tls podName:3dbb737f-ccec-4959-af11-33486428f42f nodeName:}" failed. No retries permitted until 2026-11-03 19:27:52.221054363 +0000 UTC m=+2606.193615529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-maas-api-tls") pod "maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" (UID: "3dbb737f-ccec-4959-af11-33486428f42f") : secret "maas-api-e2e-inf-b-22ed3b-serving-cert" not found Nov 03 19:27:51.721534 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.721509 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbb737f-ccec-4959-af11-33486428f42f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:51.723289 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.723261 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-metrics-certs\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:51.798239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:51.798159 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xltv\" (UniqueName: \"kubernetes.io/projected/3dbb737f-ccec-4959-af11-33486428f42f-kube-api-access-5xltv\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:52.236696 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:52.236657 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-maas-api-tls\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:52.239204 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:52.239177 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-maas-api-tls\") pod \"maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:52.407554 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:52.407527 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:52.785019 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:52.784989 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2"] Nov 03 19:27:52.786175 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:27:52.786143 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbb737f_ccec_4959_af11_33486428f42f.slice/crio-a2df3e652c40aa7c5e5a01dc692bca720f9cb223108b625f4909c77920fe8add WatchSource:0}: Error finding container a2df3e652c40aa7c5e5a01dc692bca720f9cb223108b625f4909c77920fe8add: Status 404 returned error can't find the container with id a2df3e652c40aa7c5e5a01dc692bca720f9cb223108b625f4909c77920fe8add Nov 03 19:27:52.787808 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:52.787786 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:27:53.391624 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:53.391585 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" event={"ID":"3dbb737f-ccec-4959-af11-33486428f42f","Type":"ContainerStarted","Data":"a2df3e652c40aa7c5e5a01dc692bca720f9cb223108b625f4909c77920fe8add"} Nov 03 19:27:54.400105 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:54.400070 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" event={"ID":"3dbb737f-ccec-4959-af11-33486428f42f","Type":"ContainerStarted","Data":"2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e"} Nov 03 19:27:54.400626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:54.400203 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:27:54.512611 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:27:54.512559 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" podStartSLOduration=2.562520396 podStartE2EDuration="3.512545517s" totalImagesPullingTime="950.025121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:54.510234099 +0000 UTC m=+2608.482795281" watchObservedRunningTime="2026-11-03 19:27:54.512545517 +0000 UTC m=+2608.485106698" Nov 03 19:28:00.414995 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:00.414964 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:28:11.304065 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:28:11.304023 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:28:27.655618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:27.655522 5019 generic.go:356] "Generic (PLEG): container finished" podID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerID="7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93" exitCode=0 Nov 03 19:28:27.655618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:27.655600 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" event={"ID":"0629a8f5-b876-46b6-bb0f-5d73ae38a489","Type":"ContainerDied","Data":"7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93"} Nov 03 19:28:27.656091 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:27.656061 5019 scope.go:122] "RemoveContainer" containerID="7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93" Nov 03 19:28:27.657712 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:27.657675 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:28:27.657823 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:27.657755 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:28:27.657823 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:27.657788 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:28:28.694639 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:28.694601 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" event={"ID":"0629a8f5-b876-46b6-bb0f-5d73ae38a489","Type":"ContainerStarted","Data":"46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea"} Nov 03 19:28:28.750931 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:28.750900 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:28:28.751896 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:28.751869 5019 patch_prober.go:29] interesting pod/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:28:28.752024 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:28.751928 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:28:29.751012 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:29.750984 5019 patch_prober.go:29] interesting pod/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:28:29.751429 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:29.751032 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:28:30.758351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:30.753632 5019 patch_prober.go:29] interesting pod/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:28:30.758351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:30.753696 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:28:31.751431 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:31.751393 5019 patch_prober.go:29] interesting pod/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:28:31.751631 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:31.751451 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:28:32.751652 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:32.751616 5019 patch_prober.go:29] interesting pod/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:28:32.752075 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:32.751668 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:28:32.881351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:32.870438 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:28:33.751745 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:33.751706 5019 patch_prober.go:29] interesting pod/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:28:33.752141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:33.751773 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:28:34.756454 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:34.756420 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:28:34.757525 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:34.757502 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:28:37.937104 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:37.936847 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5"] Nov 03 19:28:37.937104 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:37.937083 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" containerID="cri-o://d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729" gracePeriod=30 Nov 03 19:28:41.300025 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:28:41.299986 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:28:43.222364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.222291 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 19:28:43.309508 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.309477 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-kube-api-access-zhm9z\" (UniqueName: \"kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-kube-api-access-zhm9z\") pod \"a5b59638-64eb-4da2-93aa-aa8f37354323\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " Nov 03 19:28:43.309712 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.309530 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-podinfo\") pod \"a5b59638-64eb-4da2-93aa-aa8f37354323\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " Nov 03 19:28:43.309712 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.309581 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-certs\") pod \"a5b59638-64eb-4da2-93aa-aa8f37354323\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " Nov 03 19:28:43.309712 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.309604 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-envoy\") pod \"a5b59638-64eb-4da2-93aa-aa8f37354323\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " Nov 03 19:28:43.309899 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.309710 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-token\" (UniqueName: \"kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-token\") pod \"a5b59638-64eb-4da2-93aa-aa8f37354323\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " Nov 03 19:28:43.309899 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.309743 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-data\") pod \"a5b59638-64eb-4da2-93aa-aa8f37354323\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " Nov 03 19:28:43.309899 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.309807 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b59638-64eb-4da2-93aa-aa8f37354323-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b59638-64eb-4da2-93aa-aa8f37354323-istiod-ca-cert\") pod \"a5b59638-64eb-4da2-93aa-aa8f37354323\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " Nov 03 19:28:43.309899 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.309869 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-socket\") pod \"a5b59638-64eb-4da2-93aa-aa8f37354323\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " Nov 03 19:28:43.310111 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.309903 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-certs" pod "a5b59638-64eb-4da2-93aa-aa8f37354323" (UID: "a5b59638-64eb-4da2-93aa-aa8f37354323"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:43.310111 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.309944 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-credential-socket\") pod \"a5b59638-64eb-4da2-93aa-aa8f37354323\" (UID: \"a5b59638-64eb-4da2-93aa-aa8f37354323\") " Nov 03 19:28:43.310224 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.310120 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-data" pod "a5b59638-64eb-4da2-93aa-aa8f37354323" (UID: "a5b59638-64eb-4da2-93aa-aa8f37354323"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:43.310426 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.310404 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-credential-socket" pod "a5b59638-64eb-4da2-93aa-aa8f37354323" (UID: "a5b59638-64eb-4da2-93aa-aa8f37354323"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:43.310549 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.310519 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b59638-64eb-4da2-93aa-aa8f37354323-istiod-ca-cert" pod "a5b59638-64eb-4da2-93aa-aa8f37354323" (UID: "a5b59638-64eb-4da2-93aa-aa8f37354323"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:28:43.310746 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.310725 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:43.310827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.310752 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:43.310827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.310769 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b59638-64eb-4da2-93aa-aa8f37354323-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:43.310827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.310785 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:43.310990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.310894 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-socket" pod "a5b59638-64eb-4da2-93aa-aa8f37354323" (UID: "a5b59638-64eb-4da2-93aa-aa8f37354323"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:43.312195 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.312165 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-envoy" pod "a5b59638-64eb-4da2-93aa-aa8f37354323" (UID: "a5b59638-64eb-4da2-93aa-aa8f37354323"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:28:43.312467 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.312448 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-token" pod "a5b59638-64eb-4da2-93aa-aa8f37354323" (UID: "a5b59638-64eb-4da2-93aa-aa8f37354323"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:28:43.312559 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.312505 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-podinfo" pod "a5b59638-64eb-4da2-93aa-aa8f37354323" (UID: "a5b59638-64eb-4da2-93aa-aa8f37354323"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:28:43.312616 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.312599 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-kube-api-access-zhm9z" pod "a5b59638-64eb-4da2-93aa-aa8f37354323" (UID: "a5b59638-64eb-4da2-93aa-aa8f37354323"). InnerVolumeSpecName "kube-api-access-zhm9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:28:43.411990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.411955 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhm9z\" (UniqueName: \"kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-kube-api-access-zhm9z\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:43.411990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.411983 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:43.411990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.411992 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:43.412284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.412003 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b59638-64eb-4da2-93aa-aa8f37354323-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:43.412284 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.412012 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b59638-64eb-4da2-93aa-aa8f37354323-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:43.825357 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.825296 5019 generic.go:356] "Generic (PLEG): container finished" podID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerID="d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729" exitCode=0 Nov 03 19:28:43.825545 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.825373 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" Nov 03 19:28:43.825545 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.825371 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" event={"ID":"a5b59638-64eb-4da2-93aa-aa8f37354323","Type":"ContainerDied","Data":"d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729"} Nov 03 19:28:43.825545 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.825451 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5" event={"ID":"a5b59638-64eb-4da2-93aa-aa8f37354323","Type":"ContainerDied","Data":"665dda3f773949e952995ee5216259b5cbfe83e5bcb6ff2f70dad64f822e57c4"} Nov 03 19:28:43.825545 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.825482 5019 scope.go:122] "RemoveContainer" containerID="d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729" Nov 03 19:28:43.836520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.836501 5019 scope.go:122] "RemoveContainer" containerID="9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4" Nov 03 19:28:43.846052 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.846034 5019 scope.go:122] "RemoveContainer" containerID="d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729" Nov 03 19:28:43.846358 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:28:43.846334 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729\": container with ID starting with d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729 not found: ID does not exist" containerID="d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729" Nov 03 19:28:43.846425 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.846370 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729"} err="failed to get container status \"d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729\": rpc error: code = NotFound desc = could not find container \"d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729\": container with ID starting with d39376bacbdfb3217eda6a6342a297d61454ea78982620429130e58fc3deb729 not found: ID does not exist" Nov 03 19:28:43.846425 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.846394 5019 scope.go:122] "RemoveContainer" containerID="9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4" Nov 03 19:28:43.846673 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:28:43.846657 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4\": container with ID starting with 9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4 not found: ID does not exist" containerID="9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4" Nov 03 19:28:43.846725 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.846679 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4"} err="failed to get container status \"9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4\": rpc error: code = NotFound desc = could not find container \"9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4\": container with ID starting with 9a927ad72250c3d569485e7251fccee683fc44bf91b9fb068135c8c91de6ebb4 not found: ID does not exist" Nov 03 19:28:43.980430 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:43.980399 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5"] Nov 03 19:28:44.026634 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:44.026596 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-vt6z5"] Nov 03 19:28:44.152019 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:28:44.151944 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" path="/var/lib/kubelet/pods/a5b59638-64eb-4da2-93aa-aa8f37354323/volumes" Nov 03 19:29:11.300354 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:29:11.300291 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:29:29.820414 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:29.820381 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:29:29.825321 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:29.825282 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:29:41.297457 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:29:41.297407 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:29:44.833511 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:44.833479 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5"] Nov 03 19:29:44.834106 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:44.833782 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" podUID="92a817a4-2971-4470-801e-799a8f1f71e3" containerName="maas-api" containerID="cri-o://c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a" gracePeriod=30 Nov 03 19:29:45.159057 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.159028 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:29:45.275068 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.275030 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-maas-api-tls\") pod \"92a817a4-2971-4470-801e-799a8f1f71e3\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " Nov 03 19:29:45.275270 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.275131 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-metrics-certs\") pod \"92a817a4-2971-4470-801e-799a8f1f71e3\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " Nov 03 19:29:45.275395 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.275282 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a817a4-2971-4470-801e-799a8f1f71e3-kube-api-access-pp7dg\" (UniqueName: \"kubernetes.io/projected/92a817a4-2971-4470-801e-799a8f1f71e3-kube-api-access-pp7dg\") pod \"92a817a4-2971-4470-801e-799a8f1f71e3\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " Nov 03 19:29:45.275395 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.275343 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a817a4-2971-4470-801e-799a8f1f71e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a817a4-2971-4470-801e-799a8f1f71e3-openshift-service-ca\") pod \"92a817a4-2971-4470-801e-799a8f1f71e3\" (UID: \"92a817a4-2971-4470-801e-799a8f1f71e3\") " Nov 03 19:29:45.280872 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.280840 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a817a4-2971-4470-801e-799a8f1f71e3-openshift-service-ca" pod "92a817a4-2971-4470-801e-799a8f1f71e3" (UID: "92a817a4-2971-4470-801e-799a8f1f71e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:29:45.281285 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.281257 5019 generic.go:356] "Generic (PLEG): container finished" podID="92a817a4-2971-4470-801e-799a8f1f71e3" containerID="c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a" exitCode=0 Nov 03 19:29:45.281462 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.281444 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" Nov 03 19:29:45.281542 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.281503 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" event={"ID":"92a817a4-2971-4470-801e-799a8f1f71e3","Type":"ContainerDied","Data":"c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a"} Nov 03 19:29:45.281613 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.281581 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5" event={"ID":"92a817a4-2971-4470-801e-799a8f1f71e3","Type":"ContainerDied","Data":"42f3f25664123ae20090f276b28234d0ed1393c81bd66c4398b6c162186ed996"} Nov 03 19:29:45.281671 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.281634 5019 scope.go:122] "RemoveContainer" containerID="c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a" Nov 03 19:29:45.283486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.283459 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-metrics-certs" pod "92a817a4-2971-4470-801e-799a8f1f71e3" (UID: "92a817a4-2971-4470-801e-799a8f1f71e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:29:45.284701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.284668 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a817a4-2971-4470-801e-799a8f1f71e3-kube-api-access-pp7dg" pod "92a817a4-2971-4470-801e-799a8f1f71e3" (UID: "92a817a4-2971-4470-801e-799a8f1f71e3"). InnerVolumeSpecName "kube-api-access-pp7dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:29:45.287806 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.287290 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-maas-api-tls" pod "92a817a4-2971-4470-801e-799a8f1f71e3" (UID: "92a817a4-2971-4470-801e-799a8f1f71e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:29:45.321378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.321350 5019 scope.go:122] "RemoveContainer" containerID="c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a" Nov 03 19:29:45.321804 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:29:45.321774 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a\": container with ID starting with c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a not found: ID does not exist" containerID="c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a" Nov 03 19:29:45.321873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.321820 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a"} err="failed to get container status \"c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a\": rpc error: code = NotFound desc = could not find container \"c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a\": container with ID starting with c2343ff47dce1a868e11997cb5587621fc6e0bc25a17f28f7a8e474870da589a not found: ID does not exist" Nov 03 19:29:45.381844 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.381777 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:45.381844 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.381801 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a817a4-2971-4470-801e-799a8f1f71e3-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:45.381844 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.381814 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp7dg\" (UniqueName: \"kubernetes.io/projected/92a817a4-2971-4470-801e-799a8f1f71e3-kube-api-access-pp7dg\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:45.381844 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.381825 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a817a4-2971-4470-801e-799a8f1f71e3-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:45.697564 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.697533 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5"] Nov 03 19:29:45.746357 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:45.746232 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-efe6c7-65d9897d89-47gw5"] Nov 03 19:29:46.151868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:46.151789 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a817a4-2971-4470-801e-799a8f1f71e3" path="/var/lib/kubelet/pods/92a817a4-2971-4470-801e-799a8f1f71e3/volumes" Nov 03 19:29:54.051762 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:54.051726 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg"] Nov 03 19:29:54.052156 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:54.051983 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" containerID="cri-o://06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007" gracePeriod=30 Nov 03 19:29:58.949301 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:58.949263 5019 patch_prober.go:29] interesting pod/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:29:58.949739 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:58.949373 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:29:59.340189 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.340157 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:29:59.425212 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.425178 5019 generic.go:356] "Generic (PLEG): container finished" podID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerID="06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007" exitCode=0 Nov 03 19:29:59.425464 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.425238 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" Nov 03 19:29:59.425464 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.425270 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" event={"ID":"833f3bdc-4284-4f00-8666-44d8a4a55791","Type":"ContainerDied","Data":"06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007"} Nov 03 19:29:59.425464 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.425364 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg" event={"ID":"833f3bdc-4284-4f00-8666-44d8a4a55791","Type":"ContainerDied","Data":"f1a478fe7bb1fa9d97c351ebb1d085e8dd37cb7a7469c765623fd1924ed1382d"} Nov 03 19:29:59.425464 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.425390 5019 scope.go:122] "RemoveContainer" containerID="06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007" Nov 03 19:29:59.435117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.435096 5019 scope.go:122] "RemoveContainer" containerID="41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82" Nov 03 19:29:59.444305 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.444288 5019 scope.go:122] "RemoveContainer" containerID="06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007" Nov 03 19:29:59.444588 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:29:59.444568 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007\": container with ID starting with 06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007 not found: ID does not exist" containerID="06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007" Nov 03 19:29:59.444653 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.444598 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007"} err="failed to get container status \"06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007\": rpc error: code = NotFound desc = could not find container \"06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007\": container with ID starting with 06abbbf8fa514bb0a32e346ac68ec9ea4ef477d1c67f24797ce21a6a6bb4a007 not found: ID does not exist" Nov 03 19:29:59.444653 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.444615 5019 scope.go:122] "RemoveContainer" containerID="41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82" Nov 03 19:29:59.444852 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:29:59.444832 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82\": container with ID starting with 41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82 not found: ID does not exist" containerID="41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82" Nov 03 19:29:59.444891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.444860 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82"} err="failed to get container status \"41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82\": rpc error: code = NotFound desc = could not find container \"41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82\": container with ID starting with 41898760a69dd705c37a1837a8bc24be872acd838081d9fe20a2088b032e7b82 not found: ID does not exist" Nov 03 19:29:59.485988 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.485902 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-credential-socket\") pod \"833f3bdc-4284-4f00-8666-44d8a4a55791\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " Nov 03 19:29:59.486164 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.486063 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-socket\") pod \"833f3bdc-4284-4f00-8666-44d8a4a55791\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " Nov 03 19:29:59.486164 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.486104 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/833f3bdc-4284-4f00-8666-44d8a4a55791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833f3bdc-4284-4f00-8666-44d8a4a55791-istiod-ca-cert\") pod \"833f3bdc-4284-4f00-8666-44d8a4a55791\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " Nov 03 19:29:59.486293 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.486193 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-token\" (UniqueName: \"kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-token\") pod \"833f3bdc-4284-4f00-8666-44d8a4a55791\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " Nov 03 19:29:59.486293 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.486211 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-credential-socket" pod "833f3bdc-4284-4f00-8666-44d8a4a55791" (UID: "833f3bdc-4284-4f00-8666-44d8a4a55791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:29:59.486293 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.486223 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-podinfo\") pod \"833f3bdc-4284-4f00-8666-44d8a4a55791\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " Nov 03 19:29:59.486293 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.486256 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-envoy\") pod \"833f3bdc-4284-4f00-8666-44d8a4a55791\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " Nov 03 19:29:59.486293 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.486285 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-kube-api-access-mrhcn\" (UniqueName: \"kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-kube-api-access-mrhcn\") pod \"833f3bdc-4284-4f00-8666-44d8a4a55791\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " Nov 03 19:29:59.486588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.486389 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-socket" pod "833f3bdc-4284-4f00-8666-44d8a4a55791" (UID: "833f3bdc-4284-4f00-8666-44d8a4a55791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:29:59.486588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.486411 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-certs\") pod \"833f3bdc-4284-4f00-8666-44d8a4a55791\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " Nov 03 19:29:59.486588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.486457 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-data\") pod \"833f3bdc-4284-4f00-8666-44d8a4a55791\" (UID: \"833f3bdc-4284-4f00-8666-44d8a4a55791\") " Nov 03 19:29:59.486588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.486539 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833f3bdc-4284-4f00-8666-44d8a4a55791-istiod-ca-cert" pod "833f3bdc-4284-4f00-8666-44d8a4a55791" (UID: "833f3bdc-4284-4f00-8666-44d8a4a55791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:29:59.486987 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.486961 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-certs" pod "833f3bdc-4284-4f00-8666-44d8a4a55791" (UID: "833f3bdc-4284-4f00-8666-44d8a4a55791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:29:59.487118 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.487096 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:59.487201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.487151 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:59.487201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.487154 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-data" pod "833f3bdc-4284-4f00-8666-44d8a4a55791" (UID: "833f3bdc-4284-4f00-8666-44d8a4a55791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:29:59.487201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.487167 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833f3bdc-4284-4f00-8666-44d8a4a55791-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:59.487201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.487182 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:59.490375 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.489271 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-podinfo" pod "833f3bdc-4284-4f00-8666-44d8a4a55791" (UID: "833f3bdc-4284-4f00-8666-44d8a4a55791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:29:59.490375 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.489422 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-kube-api-access-mrhcn" pod "833f3bdc-4284-4f00-8666-44d8a4a55791" (UID: "833f3bdc-4284-4f00-8666-44d8a4a55791"). InnerVolumeSpecName "kube-api-access-mrhcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:29:59.490375 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.489628 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-envoy" pod "833f3bdc-4284-4f00-8666-44d8a4a55791" (UID: "833f3bdc-4284-4f00-8666-44d8a4a55791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:29:59.490829 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.490806 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-token" pod "833f3bdc-4284-4f00-8666-44d8a4a55791" (UID: "833f3bdc-4284-4f00-8666-44d8a4a55791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:29:59.588958 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.588921 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:59.588958 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.588960 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:59.589189 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.588977 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:59.589189 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.588992 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrhcn\" (UniqueName: \"kubernetes.io/projected/833f3bdc-4284-4f00-8666-44d8a4a55791-kube-api-access-mrhcn\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:59.589189 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.589006 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833f3bdc-4284-4f00-8666-44d8a4a55791-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:59.840839 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.840750 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg"] Nov 03 19:29:59.887854 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:29:59.887816 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-efe6c7-openshift-default-6895b99cdc-9jftg"] Nov 03 19:30:00.153008 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:00.152913 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" path="/var/lib/kubelet/pods/833f3bdc-4284-4f00-8666-44d8a4a55791/volumes" Nov 03 19:30:11.296829 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:30:11.296734 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:30:14.988562 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:14.988528 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9"] Nov 03 19:30:14.989041 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:14.988762 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" podUID="f448ccd6-0521-480f-8c09-04d31e3858f2" containerName="maas-api" containerID="cri-o://63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a" gracePeriod=30 Nov 03 19:30:15.347031 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.347004 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:30:15.369749 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.369721 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-metrics-certs\") pod \"f448ccd6-0521-480f-8c09-04d31e3858f2\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " Nov 03 19:30:15.369894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.369793 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f448ccd6-0521-480f-8c09-04d31e3858f2-kube-api-access-wst6j\" (UniqueName: \"kubernetes.io/projected/f448ccd6-0521-480f-8c09-04d31e3858f2-kube-api-access-wst6j\") pod \"f448ccd6-0521-480f-8c09-04d31e3858f2\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " Nov 03 19:30:15.369997 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.369983 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-maas-api-tls\") pod \"f448ccd6-0521-480f-8c09-04d31e3858f2\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " Nov 03 19:30:15.370058 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.370015 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f448ccd6-0521-480f-8c09-04d31e3858f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f448ccd6-0521-480f-8c09-04d31e3858f2-openshift-service-ca\") pod \"f448ccd6-0521-480f-8c09-04d31e3858f2\" (UID: \"f448ccd6-0521-480f-8c09-04d31e3858f2\") " Nov 03 19:30:15.370461 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.370430 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f448ccd6-0521-480f-8c09-04d31e3858f2-openshift-service-ca" pod "f448ccd6-0521-480f-8c09-04d31e3858f2" (UID: "f448ccd6-0521-480f-8c09-04d31e3858f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:30:15.370829 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.370801 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f448ccd6-0521-480f-8c09-04d31e3858f2-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:15.371886 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.371858 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-metrics-certs" pod "f448ccd6-0521-480f-8c09-04d31e3858f2" (UID: "f448ccd6-0521-480f-8c09-04d31e3858f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:30:15.371979 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.371938 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-maas-api-tls" pod "f448ccd6-0521-480f-8c09-04d31e3858f2" (UID: "f448ccd6-0521-480f-8c09-04d31e3858f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:30:15.372069 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.372052 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f448ccd6-0521-480f-8c09-04d31e3858f2-kube-api-access-wst6j" pod "f448ccd6-0521-480f-8c09-04d31e3858f2" (UID: "f448ccd6-0521-480f-8c09-04d31e3858f2"). InnerVolumeSpecName "kube-api-access-wst6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:15.517389 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.476000 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:15.517389 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.476038 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f448ccd6-0521-480f-8c09-04d31e3858f2-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:15.517389 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.476054 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wst6j\" (UniqueName: \"kubernetes.io/projected/f448ccd6-0521-480f-8c09-04d31e3858f2-kube-api-access-wst6j\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:15.607903 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.607822 5019 generic.go:356] "Generic (PLEG): container finished" podID="f448ccd6-0521-480f-8c09-04d31e3858f2" containerID="63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a" exitCode=0 Nov 03 19:30:15.607903 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.607869 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" Nov 03 19:30:15.608154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.607909 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" event={"ID":"f448ccd6-0521-480f-8c09-04d31e3858f2","Type":"ContainerDied","Data":"63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a"} Nov 03 19:30:15.608154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.607976 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9" event={"ID":"f448ccd6-0521-480f-8c09-04d31e3858f2","Type":"ContainerDied","Data":"32f7a670c5bc3626ed5329d36d6d001e28775fb7d91bf80290675268c04beca7"} Nov 03 19:30:15.608154 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.608005 5019 scope.go:122] "RemoveContainer" containerID="63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a" Nov 03 19:30:15.623553 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.623501 5019 scope.go:122] "RemoveContainer" containerID="63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a" Nov 03 19:30:15.623946 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:30:15.623908 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a\": container with ID starting with 63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a not found: ID does not exist" containerID="63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a" Nov 03 19:30:15.624030 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.623971 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a"} err="failed to get container status \"63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a\": rpc error: code = NotFound desc = could not find container \"63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a\": container with ID starting with 63a89143b5446c945d5fadbd93ad73134f1de487b8f78c106ed990bcb165ae4a not found: ID does not exist" Nov 03 19:30:15.762481 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.762447 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9"] Nov 03 19:30:15.766424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:15.766402 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1410ed-8555546f4d-jkxz9"] Nov 03 19:30:16.151434 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:16.151400 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f448ccd6-0521-480f-8c09-04d31e3858f2" path="/var/lib/kubelet/pods/f448ccd6-0521-480f-8c09-04d31e3858f2/volumes" Nov 03 19:30:23.869704 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:23.869671 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8"] Nov 03 19:30:23.870206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:23.869876 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" podUID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" containerName="istio-proxy" containerID="cri-o://5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d" gracePeriod=30 Nov 03 19:30:28.013349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:28.013282 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:30:28.013738 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:28.013394 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" podUID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:30:29.132247 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.132190 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:30:29.255884 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.255843 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-data\") pod \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " Nov 03 19:30:29.256076 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.255913 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-token\" (UniqueName: \"kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-token\") pod \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " Nov 03 19:30:29.256076 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.255967 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-envoy\") pod \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " Nov 03 19:30:29.256076 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.255991 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-kube-api-access-bnps7\" (UniqueName: \"kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-kube-api-access-bnps7\") pod \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " Nov 03 19:30:29.256266 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.256074 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-podinfo\") pod \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " Nov 03 19:30:29.256266 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.256124 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-data" pod "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" (UID: "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:30:29.256266 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.256145 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-credential-socket\") pod \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " Nov 03 19:30:29.256266 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.256219 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-certs\") pod \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " Nov 03 19:30:29.256528 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.256367 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istiod-ca-cert\") pod \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " Nov 03 19:30:29.256528 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.256403 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-socket\") pod \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\" (UID: \"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3\") " Nov 03 19:30:29.256528 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.256486 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-credential-socket" pod "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" (UID: "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:30:29.256757 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.256737 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-certs" pod "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" (UID: "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:30:29.257148 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.256879 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istiod-ca-cert" pod "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" (UID: "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:30:29.257148 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.257079 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-socket" pod "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" (UID: "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:30:29.257365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.257177 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.257365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.257199 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.257365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.257212 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.257365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.257225 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.258450 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.258424 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-token" pod "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" (UID: "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:29.258606 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.258535 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-kube-api-access-bnps7" pod "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" (UID: "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3"). InnerVolumeSpecName "kube-api-access-bnps7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:29.258866 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.258837 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-envoy" pod "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" (UID: "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:30:29.258957 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.258846 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-podinfo" pod "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" (UID: "ea234ecc-29a0-4bdf-9784-44d3ea7f25e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:30:29.358705 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.358664 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.358705 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.358693 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.358705 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.358705 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.358705 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.358714 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnps7\" (UniqueName: \"kubernetes.io/projected/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-kube-api-access-bnps7\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.359067 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.358724 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.698752 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.698719 5019 generic.go:356] "Generic (PLEG): container finished" podID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" containerID="5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d" exitCode=0 Nov 03 19:30:29.698940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.698749 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" event={"ID":"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3","Type":"ContainerDied","Data":"5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d"} Nov 03 19:30:29.698940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.698805 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" event={"ID":"ea234ecc-29a0-4bdf-9784-44d3ea7f25e3","Type":"ContainerDied","Data":"b5ef35f915a6b6d17adf1219cba4c488e10055a3f021434aed14f72df1793e1e"} Nov 03 19:30:29.698940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.698762 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8" Nov 03 19:30:29.698940 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.698828 5019 scope.go:122] "RemoveContainer" containerID="5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d" Nov 03 19:30:29.709901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.709882 5019 scope.go:122] "RemoveContainer" containerID="5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d" Nov 03 19:30:29.710160 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:30:29.710142 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d\": container with ID starting with 5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d not found: ID does not exist" containerID="5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d" Nov 03 19:30:29.710207 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.710169 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d"} err="failed to get container status \"5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d\": rpc error: code = NotFound desc = could not find container \"5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d\": container with ID starting with 5c66b9bf39ac556649f0cee356eb7c7ee53c0c1f63b0fb61eb26bb4089a1665d not found: ID does not exist" Nov 03 19:30:29.854669 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.854638 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8"] Nov 03 19:30:29.865145 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:29.865113 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1410ed-openshift-default-779877778f-h6wp8"] Nov 03 19:30:30.152372 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:30.152318 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" path="/var/lib/kubelet/pods/ea234ecc-29a0-4bdf-9784-44d3ea7f25e3/volumes" Nov 03 19:30:41.297599 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:30:41.297561 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:30:58.996366 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.996318 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f"] Nov 03 19:30:58.996977 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.996961 5019 cpu_manager.go:414] "containerMap: removing container" podUID="92a817a4-2971-4470-801e-799a8f1f71e3" containerName="maas-api" Nov 03 19:30:58.997032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.996978 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a817a4-2971-4470-801e-799a8f1f71e3" containerName="maas-api" Nov 03 19:30:58.997032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.996993 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" Nov 03 19:30:58.997032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.996998 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" Nov 03 19:30:58.997032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997014 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" containerName="istio-proxy" Nov 03 19:30:58.997032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997020 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" containerName="istio-proxy" Nov 03 19:30:58.997032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997029 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f448ccd6-0521-480f-8c09-04d31e3858f2" containerName="maas-api" Nov 03 19:30:58.997032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997034 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f448ccd6-0521-480f-8c09-04d31e3858f2" containerName="maas-api" Nov 03 19:30:58.997257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997048 5019 cpu_manager.go:414] "containerMap: removing container" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" Nov 03 19:30:58.997257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997052 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" Nov 03 19:30:58.997257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997064 5019 cpu_manager.go:414] "containerMap: removing container" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" Nov 03 19:30:58.997257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997068 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" Nov 03 19:30:58.997257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997082 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" Nov 03 19:30:58.997257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997089 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" Nov 03 19:30:58.997257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997180 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" Nov 03 19:30:58.997257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997190 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="833f3bdc-4284-4f00-8666-44d8a4a55791" containerName="istio-proxy" Nov 03 19:30:58.997257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997197 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" Nov 03 19:30:58.997257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997207 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a817a4-2971-4470-801e-799a8f1f71e3" containerName="maas-api" Nov 03 19:30:58.997257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997214 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea234ecc-29a0-4bdf-9784-44d3ea7f25e3" containerName="istio-proxy" Nov 03 19:30:58.997257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:58.997220 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f448ccd6-0521-480f-8c09-04d31e3858f2" containerName="maas-api" Nov 03 19:30:59.000696 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.000678 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.049603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.049581 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fcb646-serving-cert\"" Nov 03 19:30:59.100544 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.100516 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f"] Nov 03 19:30:59.181245 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.181203 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-maas-api-tls\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.181442 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.181267 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58ed2b0-024a-48af-ab22-213400acc765-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.181442 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.181381 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trfqz\" (UniqueName: \"kubernetes.io/projected/c58ed2b0-024a-48af-ab22-213400acc765-kube-api-access-trfqz\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.181526 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.181502 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-metrics-certs\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.293620 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.293523 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-metrics-certs\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.293812 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.293722 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-maas-api-tls\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.293812 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.293773 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58ed2b0-024a-48af-ab22-213400acc765-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.293812 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.293802 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trfqz\" (UniqueName: \"kubernetes.io/projected/c58ed2b0-024a-48af-ab22-213400acc765-kube-api-access-trfqz\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.296743 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.296712 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58ed2b0-024a-48af-ab22-213400acc765-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.297433 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.297404 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-metrics-certs\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.297918 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:30:59.295436 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-serving-cert: secret "maas-api-e2e-inf-a-fcb646-serving-cert" not found Nov 03 19:30:59.298006 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:30:59.297993 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-maas-api-tls podName:c58ed2b0-024a-48af-ab22-213400acc765 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:59.797971576 +0000 UTC m=+2793.770532774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-maas-api-tls") pod "maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" (UID: "c58ed2b0-024a-48af-ab22-213400acc765") : secret "maas-api-e2e-inf-a-fcb646-serving-cert" not found Nov 03 19:30:59.352069 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.352041 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trfqz\" (UniqueName: \"kubernetes.io/projected/c58ed2b0-024a-48af-ab22-213400acc765-kube-api-access-trfqz\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.805624 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.805579 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-maas-api-tls\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.807967 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.807940 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-maas-api-tls\") pod \"maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:30:59.914409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:30:59.914367 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:31:00.287395 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:00.287370 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f"] Nov 03 19:31:00.289588 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:31:00.289555 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58ed2b0_024a_48af_ab22_213400acc765.slice/crio-919101af7e0a89832a68c3c19e5a979b7f4a9a4006910778dd81874704062cd9 WatchSource:0}: Error finding container 919101af7e0a89832a68c3c19e5a979b7f4a9a4006910778dd81874704062cd9: Status 404 returned error can't find the container with id 919101af7e0a89832a68c3c19e5a979b7f4a9a4006910778dd81874704062cd9 Nov 03 19:31:00.912700 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:00.912671 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" event={"ID":"c58ed2b0-024a-48af-ab22-213400acc765","Type":"ContainerStarted","Data":"919101af7e0a89832a68c3c19e5a979b7f4a9a4006910778dd81874704062cd9"} Nov 03 19:31:01.920633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:01.920595 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" event={"ID":"c58ed2b0-024a-48af-ab22-213400acc765","Type":"ContainerStarted","Data":"e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee"} Nov 03 19:31:01.921058 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:01.920729 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:31:02.033340 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:02.033269 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" podStartSLOduration=3.165696326 podStartE2EDuration="4.033256368s" totalImagesPullingTime="867.560042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:02.031729781 +0000 UTC m=+2796.004290963" watchObservedRunningTime="2026-11-03 19:31:02.033256368 +0000 UTC m=+2796.005817549" Nov 03 19:31:07.938089 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:07.938061 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:31:11.298172 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:31:11.298125 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:31:16.957305 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:16.957265 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf"] Nov 03 19:31:16.957871 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:16.957538 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" podUID="f4773c82-6fb1-4108-aa54-a6e5635a847b" containerName="maas-api" containerID="cri-o://97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023" gracePeriod=30 Nov 03 19:31:17.260113 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.260081 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:31:17.296800 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.296770 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4773c82-6fb1-4108-aa54-a6e5635a847b-kube-api-access-r2vt2\" (UniqueName: \"kubernetes.io/projected/f4773c82-6fb1-4108-aa54-a6e5635a847b-kube-api-access-r2vt2\") pod \"f4773c82-6fb1-4108-aa54-a6e5635a847b\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " Nov 03 19:31:17.296948 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.296818 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4773c82-6fb1-4108-aa54-a6e5635a847b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4773c82-6fb1-4108-aa54-a6e5635a847b-openshift-service-ca\") pod \"f4773c82-6fb1-4108-aa54-a6e5635a847b\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " Nov 03 19:31:17.296948 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.296910 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-metrics-certs\") pod \"f4773c82-6fb1-4108-aa54-a6e5635a847b\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " Nov 03 19:31:17.297059 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.297042 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-maas-api-tls\") pod \"f4773c82-6fb1-4108-aa54-a6e5635a847b\" (UID: \"f4773c82-6fb1-4108-aa54-a6e5635a847b\") " Nov 03 19:31:17.297265 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.297237 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4773c82-6fb1-4108-aa54-a6e5635a847b-openshift-service-ca" pod "f4773c82-6fb1-4108-aa54-a6e5635a847b" (UID: "f4773c82-6fb1-4108-aa54-a6e5635a847b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:31:17.297613 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.297589 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4773c82-6fb1-4108-aa54-a6e5635a847b-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:17.299023 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.298998 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4773c82-6fb1-4108-aa54-a6e5635a847b-kube-api-access-r2vt2" pod "f4773c82-6fb1-4108-aa54-a6e5635a847b" (UID: "f4773c82-6fb1-4108-aa54-a6e5635a847b"). InnerVolumeSpecName "kube-api-access-r2vt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:17.299023 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.299002 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-metrics-certs" pod "f4773c82-6fb1-4108-aa54-a6e5635a847b" (UID: "f4773c82-6fb1-4108-aa54-a6e5635a847b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:31:17.299166 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.299070 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-maas-api-tls" pod "f4773c82-6fb1-4108-aa54-a6e5635a847b" (UID: "f4773c82-6fb1-4108-aa54-a6e5635a847b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:31:17.398760 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.398728 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2vt2\" (UniqueName: \"kubernetes.io/projected/f4773c82-6fb1-4108-aa54-a6e5635a847b-kube-api-access-r2vt2\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:17.398760 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.398758 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:17.398760 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:17.398768 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4773c82-6fb1-4108-aa54-a6e5635a847b-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:18.033827 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:18.033796 5019 generic.go:356] "Generic (PLEG): container finished" podID="f4773c82-6fb1-4108-aa54-a6e5635a847b" containerID="97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023" exitCode=0 Nov 03 19:31:18.034368 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:18.033849 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" Nov 03 19:31:18.034368 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:18.033872 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" event={"ID":"f4773c82-6fb1-4108-aa54-a6e5635a847b","Type":"ContainerDied","Data":"97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023"} Nov 03 19:31:18.034368 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:18.033932 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf" event={"ID":"f4773c82-6fb1-4108-aa54-a6e5635a847b","Type":"ContainerDied","Data":"d035828d87822409539aea10195c2168f5c7a46faff0146be4a2f5757b2280f9"} Nov 03 19:31:18.034368 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:18.033956 5019 scope.go:122] "RemoveContainer" containerID="97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023" Nov 03 19:31:18.047234 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:18.047217 5019 scope.go:122] "RemoveContainer" containerID="97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023" Nov 03 19:31:18.048438 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:31:18.048419 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023\": container with ID starting with 97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023 not found: ID does not exist" containerID="97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023" Nov 03 19:31:18.048545 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:18.048447 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023"} err="failed to get container status \"97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023\": rpc error: code = NotFound desc = could not find container \"97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023\": container with ID starting with 97c254221548492cfe51929a1dcac261fc417a20578017cf5f0f3c6d8a000023 not found: ID does not exist" Nov 03 19:31:18.188703 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:18.188655 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf"] Nov 03 19:31:18.193016 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:18.192959 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6413ba-664f7cf4b-fmmzf"] Nov 03 19:31:20.151625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:20.151593 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4773c82-6fb1-4108-aa54-a6e5635a847b" path="/var/lib/kubelet/pods/f4773c82-6fb1-4108-aa54-a6e5635a847b/volumes" Nov 03 19:31:22.900801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:22.900747 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns"] Nov 03 19:31:22.901487 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:22.901018 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" podUID="a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf" containerName="maas-api" containerID="cri-o://130dd540b979ba3ea22b855ec48bb7ed17d56d4eac326d7b4c3c8620b867e289" gracePeriod=30 Nov 03 19:31:23.068620 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.068581 5019 generic.go:356] "Generic (PLEG): container finished" podID="a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf" containerID="130dd540b979ba3ea22b855ec48bb7ed17d56d4eac326d7b4c3c8620b867e289" exitCode=0 Nov 03 19:31:23.068620 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.068607 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" event={"ID":"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf","Type":"ContainerDied","Data":"130dd540b979ba3ea22b855ec48bb7ed17d56d4eac326d7b4c3c8620b867e289"} Nov 03 19:31:23.239305 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.239282 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:31:23.382997 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.382969 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-metrics-certs\") pod \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " Nov 03 19:31:23.383191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.383038 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-kube-api-access-9tn58\" (UniqueName: \"kubernetes.io/projected/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-kube-api-access-9tn58\") pod \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " Nov 03 19:31:23.383191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.383156 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-openshift-service-ca\") pod \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " Nov 03 19:31:23.383313 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.383249 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-maas-api-tls\") pod \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\" (UID: \"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf\") " Nov 03 19:31:23.384162 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.384131 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-openshift-service-ca" pod "a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf" (UID: "a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:31:23.386504 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.386477 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-kube-api-access-9tn58" pod "a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf" (UID: "a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf"). InnerVolumeSpecName "kube-api-access-9tn58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:23.386722 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.386700 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-maas-api-tls" pod "a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf" (UID: "a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:31:23.386789 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.386718 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-metrics-certs" pod "a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf" (UID: "a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:31:23.497749 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.489932 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:23.497749 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.489986 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tn58\" (UniqueName: \"kubernetes.io/projected/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-kube-api-access-9tn58\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:23.497749 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.490002 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:23.497749 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:23.490016 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:24.076508 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:24.076460 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" event={"ID":"a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf","Type":"ContainerDied","Data":"f9188ca34ddc7a1c51ef72f6a07d3c2d67f319c4cdadddff23ec231582b21561"} Nov 03 19:31:24.077028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:24.076538 5019 scope.go:122] "RemoveContainer" containerID="130dd540b979ba3ea22b855ec48bb7ed17d56d4eac326d7b4c3c8620b867e289" Nov 03 19:31:24.077028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:24.076469 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns" Nov 03 19:31:24.236389 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:24.236359 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns"] Nov 03 19:31:24.282906 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:24.282876 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-332dac-8dffdd8db-74pns"] Nov 03 19:31:26.179183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:26.179141 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf" path="/var/lib/kubelet/pods/a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf/volumes" Nov 03 19:31:31.938717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:31.938673 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm"] Nov 03 19:31:31.939188 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:31.939020 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" podUID="11d500e8-f34b-4fe8-ad68-363552ebd6f6" containerName="istio-proxy" containerID="cri-o://447f5a726744a69bc2837d6f00d08164115bdb24a3fa0ad4dd2f0803c1ac348b" gracePeriod=30 Nov 03 19:31:37.219519 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.219444 5019 generic.go:356] "Generic (PLEG): container finished" podID="11d500e8-f34b-4fe8-ad68-363552ebd6f6" containerID="447f5a726744a69bc2837d6f00d08164115bdb24a3fa0ad4dd2f0803c1ac348b" exitCode=0 Nov 03 19:31:37.219863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.219522 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" event={"ID":"11d500e8-f34b-4fe8-ad68-363552ebd6f6","Type":"ContainerDied","Data":"447f5a726744a69bc2837d6f00d08164115bdb24a3fa0ad4dd2f0803c1ac348b"} Nov 03 19:31:37.219863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.219598 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" event={"ID":"11d500e8-f34b-4fe8-ad68-363552ebd6f6","Type":"ContainerDied","Data":"4adbcacd5abb64d239616d51e405790cc2a8f884eca91029b8d387dd01a09729"} Nov 03 19:31:37.219863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.219629 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4adbcacd5abb64d239616d51e405790cc2a8f884eca91029b8d387dd01a09729" Nov 03 19:31:37.230944 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.230923 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:31:37.382611 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.382576 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-podinfo\") pod \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " Nov 03 19:31:37.382828 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.382653 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-data\") pod \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " Nov 03 19:31:37.382828 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.382732 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-token\" (UniqueName: \"kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-token\") pod \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " Nov 03 19:31:37.382828 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.382804 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-certs\") pod \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " Nov 03 19:31:37.383035 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.382854 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-credential-socket\") pod \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " Nov 03 19:31:37.383035 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.382969 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istiod-ca-cert\") pod \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " Nov 03 19:31:37.383035 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.382999 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-socket\") pod \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " Nov 03 19:31:37.383202 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.383059 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-data" pod "11d500e8-f34b-4fe8-ad68-363552ebd6f6" (UID: "11d500e8-f34b-4fe8-ad68-363552ebd6f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:37.383202 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.383072 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-envoy\") pod \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " Nov 03 19:31:37.383202 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.383132 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-kube-api-access-dc6cs\" (UniqueName: \"kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-kube-api-access-dc6cs\") pod \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\" (UID: \"11d500e8-f34b-4fe8-ad68-363552ebd6f6\") " Nov 03 19:31:37.383801 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.383775 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.384561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.384059 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-certs" pod "11d500e8-f34b-4fe8-ad68-363552ebd6f6" (UID: "11d500e8-f34b-4fe8-ad68-363552ebd6f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:37.384561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.384084 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-credential-socket" pod "11d500e8-f34b-4fe8-ad68-363552ebd6f6" (UID: "11d500e8-f34b-4fe8-ad68-363552ebd6f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:37.384561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.384176 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istiod-ca-cert" pod "11d500e8-f34b-4fe8-ad68-363552ebd6f6" (UID: "11d500e8-f34b-4fe8-ad68-363552ebd6f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:31:37.384561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.384310 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-socket" pod "11d500e8-f34b-4fe8-ad68-363552ebd6f6" (UID: "11d500e8-f34b-4fe8-ad68-363552ebd6f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:37.386674 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.386654 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-envoy" pod "11d500e8-f34b-4fe8-ad68-363552ebd6f6" (UID: "11d500e8-f34b-4fe8-ad68-363552ebd6f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:31:37.386821 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.386806 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-kube-api-access-dc6cs" pod "11d500e8-f34b-4fe8-ad68-363552ebd6f6" (UID: "11d500e8-f34b-4fe8-ad68-363552ebd6f6"). InnerVolumeSpecName "kube-api-access-dc6cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:37.386984 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.386963 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-token" pod "11d500e8-f34b-4fe8-ad68-363552ebd6f6" (UID: "11d500e8-f34b-4fe8-ad68-363552ebd6f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:37.394762 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.394741 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-podinfo" pod "11d500e8-f34b-4fe8-ad68-363552ebd6f6" (UID: "11d500e8-f34b-4fe8-ad68-363552ebd6f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:31:37.484845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.484765 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.484845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.484798 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.484845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.484808 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.484845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.484817 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.484845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.484826 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.484845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.484834 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc6cs\" (UniqueName: \"kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-kube-api-access-dc6cs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.484845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.484844 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:37.484845 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:37.484852 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d500e8-f34b-4fe8-ad68-363552ebd6f6-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:38.225297 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:38.225259 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm" Nov 03 19:31:38.432051 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:38.432019 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm"] Nov 03 19:31:38.478656 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:38.478621 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-332dac-openshift-default-76ff966d45-87wpm"] Nov 03 19:31:40.151342 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:31:40.151298 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d500e8-f34b-4fe8-ad68-363552ebd6f6" path="/var/lib/kubelet/pods/11d500e8-f34b-4fe8-ad68-363552ebd6f6/volumes" Nov 03 19:31:41.296815 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:31:41.296777 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:32:11.306097 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:32:11.306050 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:32:12.845501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:12.845469 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6"] Nov 03 19:32:12.845947 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:12.845688 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" podUID="b54643ca-250e-49db-b008-f7f6e15aca04" containerName="maas-api" containerID="cri-o://cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa" gracePeriod=30 Nov 03 19:32:13.181479 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.181451 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:32:13.364005 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.363973 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-maas-api-tls\") pod \"b54643ca-250e-49db-b008-f7f6e15aca04\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " Nov 03 19:32:13.364213 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.364139 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b54643ca-250e-49db-b008-f7f6e15aca04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54643ca-250e-49db-b008-f7f6e15aca04-openshift-service-ca\") pod \"b54643ca-250e-49db-b008-f7f6e15aca04\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " Nov 03 19:32:13.364213 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.364186 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b54643ca-250e-49db-b008-f7f6e15aca04-kube-api-access-mrrpd\" (UniqueName: \"kubernetes.io/projected/b54643ca-250e-49db-b008-f7f6e15aca04-kube-api-access-mrrpd\") pod \"b54643ca-250e-49db-b008-f7f6e15aca04\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " Nov 03 19:32:13.364386 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.364218 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-metrics-certs\") pod \"b54643ca-250e-49db-b008-f7f6e15aca04\" (UID: \"b54643ca-250e-49db-b008-f7f6e15aca04\") " Nov 03 19:32:13.365109 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.365082 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54643ca-250e-49db-b008-f7f6e15aca04-openshift-service-ca" pod "b54643ca-250e-49db-b008-f7f6e15aca04" (UID: "b54643ca-250e-49db-b008-f7f6e15aca04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:13.367156 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.367123 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54643ca-250e-49db-b008-f7f6e15aca04-kube-api-access-mrrpd" pod "b54643ca-250e-49db-b008-f7f6e15aca04" (UID: "b54643ca-250e-49db-b008-f7f6e15aca04"). InnerVolumeSpecName "kube-api-access-mrrpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:13.367312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.367285 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-maas-api-tls" pod "b54643ca-250e-49db-b008-f7f6e15aca04" (UID: "b54643ca-250e-49db-b008-f7f6e15aca04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:13.367462 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.367438 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-metrics-certs" pod "b54643ca-250e-49db-b008-f7f6e15aca04" (UID: "b54643ca-250e-49db-b008-f7f6e15aca04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:13.465664 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.465631 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54643ca-250e-49db-b008-f7f6e15aca04-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:13.465664 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.465659 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrrpd\" (UniqueName: \"kubernetes.io/projected/b54643ca-250e-49db-b008-f7f6e15aca04-kube-api-access-mrrpd\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:13.465664 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.465669 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:13.465963 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.465678 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b54643ca-250e-49db-b008-f7f6e15aca04-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:13.540643 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.540612 5019 generic.go:356] "Generic (PLEG): container finished" podID="b54643ca-250e-49db-b008-f7f6e15aca04" containerID="cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa" exitCode=0 Nov 03 19:32:13.540807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.540660 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" Nov 03 19:32:13.540807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.540689 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" event={"ID":"b54643ca-250e-49db-b008-f7f6e15aca04","Type":"ContainerDied","Data":"cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa"} Nov 03 19:32:13.540807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.540742 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6" event={"ID":"b54643ca-250e-49db-b008-f7f6e15aca04","Type":"ContainerDied","Data":"eec6f509649a6ccb2573348335f0e642fed1a77ca71e71e3c1ce31e23505a820"} Nov 03 19:32:13.540807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.540765 5019 scope.go:122] "RemoveContainer" containerID="cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa" Nov 03 19:32:13.554136 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.554069 5019 scope.go:122] "RemoveContainer" containerID="cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa" Nov 03 19:32:13.554498 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:32:13.554477 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa\": container with ID starting with cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa not found: ID does not exist" containerID="cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa" Nov 03 19:32:13.554593 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.554506 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa"} err="failed to get container status \"cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa\": rpc error: code = NotFound desc = could not find container \"cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa\": container with ID starting with cf1471624ecbdcd155d71613e88ae16af1ef680307641744d2c32f39a0798efa not found: ID does not exist" Nov 03 19:32:13.697829 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.697795 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6"] Nov 03 19:32:13.742850 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:13.742780 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c3e530-7c8c9f7-r4cl6"] Nov 03 19:32:14.171262 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:14.171234 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b54643ca-250e-49db-b008-f7f6e15aca04" path="/var/lib/kubelet/pods/b54643ca-250e-49db-b008-f7f6e15aca04/volumes" Nov 03 19:32:21.963317 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:21.963285 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9"] Nov 03 19:32:21.963775 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:21.963507 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" containerID="cri-o://f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0" gracePeriod=30 Nov 03 19:32:23.802629 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:23.802594 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:32:23.803041 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:23.802660 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:32:27.236697 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.236641 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:32:27.342402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.342366 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddca4421-6d98-415e-9848-80ec181cb4ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddca4421-6d98-415e-9848-80ec181cb4ae-istiod-ca-cert\") pod \"ddca4421-6d98-415e-9848-80ec181cb4ae\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " Nov 03 19:32:27.342565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.342420 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-podinfo\") pod \"ddca4421-6d98-415e-9848-80ec181cb4ae\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " Nov 03 19:32:27.342565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.342457 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-token\" (UniqueName: \"kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-token\") pod \"ddca4421-6d98-415e-9848-80ec181cb4ae\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " Nov 03 19:32:27.342565 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.342526 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-data\") pod \"ddca4421-6d98-415e-9848-80ec181cb4ae\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " Nov 03 19:32:27.342720 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.342614 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-envoy\") pod \"ddca4421-6d98-415e-9848-80ec181cb4ae\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " Nov 03 19:32:27.342720 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.342660 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-certs\") pod \"ddca4421-6d98-415e-9848-80ec181cb4ae\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " Nov 03 19:32:27.342875 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.342720 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-socket\") pod \"ddca4421-6d98-415e-9848-80ec181cb4ae\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " Nov 03 19:32:27.342875 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.342742 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddca4421-6d98-415e-9848-80ec181cb4ae-istiod-ca-cert" pod "ddca4421-6d98-415e-9848-80ec181cb4ae" (UID: "ddca4421-6d98-415e-9848-80ec181cb4ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:27.342995 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.342895 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-data" pod "ddca4421-6d98-415e-9848-80ec181cb4ae" (UID: "ddca4421-6d98-415e-9848-80ec181cb4ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:27.342995 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.342952 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-credential-socket\") pod \"ddca4421-6d98-415e-9848-80ec181cb4ae\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " Nov 03 19:32:27.342995 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.342982 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-socket" pod "ddca4421-6d98-415e-9848-80ec181cb4ae" (UID: "ddca4421-6d98-415e-9848-80ec181cb4ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:27.343162 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.342993 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-kube-api-access-drjwr\" (UniqueName: \"kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-kube-api-access-drjwr\") pod \"ddca4421-6d98-415e-9848-80ec181cb4ae\" (UID: \"ddca4421-6d98-415e-9848-80ec181cb4ae\") " Nov 03 19:32:27.343162 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.343125 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-certs" pod "ddca4421-6d98-415e-9848-80ec181cb4ae" (UID: "ddca4421-6d98-415e-9848-80ec181cb4ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:27.343278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.343210 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-credential-socket" pod "ddca4421-6d98-415e-9848-80ec181cb4ae" (UID: "ddca4421-6d98-415e-9848-80ec181cb4ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:27.343555 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.343530 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:27.343680 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.343557 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:27.343680 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.343574 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddca4421-6d98-415e-9848-80ec181cb4ae-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:27.343680 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.343588 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:27.343680 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.343602 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:27.345277 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.345246 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-envoy" pod "ddca4421-6d98-415e-9848-80ec181cb4ae" (UID: "ddca4421-6d98-415e-9848-80ec181cb4ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:27.345277 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.345253 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-token" pod "ddca4421-6d98-415e-9848-80ec181cb4ae" (UID: "ddca4421-6d98-415e-9848-80ec181cb4ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:27.345443 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.345377 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-kube-api-access-drjwr" pod "ddca4421-6d98-415e-9848-80ec181cb4ae" (UID: "ddca4421-6d98-415e-9848-80ec181cb4ae"). InnerVolumeSpecName "kube-api-access-drjwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:27.345697 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.345677 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-podinfo" pod "ddca4421-6d98-415e-9848-80ec181cb4ae" (UID: "ddca4421-6d98-415e-9848-80ec181cb4ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:32:27.444810 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.444768 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drjwr\" (UniqueName: \"kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-kube-api-access-drjwr\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:27.444810 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.444809 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:27.445018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.444826 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:27.445018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.444842 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddca4421-6d98-415e-9848-80ec181cb4ae-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:27.666523 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.666419 5019 generic.go:356] "Generic (PLEG): container finished" podID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerID="f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0" exitCode=0 Nov 03 19:32:27.666523 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.666486 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" Nov 03 19:32:27.666765 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.666523 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" event={"ID":"ddca4421-6d98-415e-9848-80ec181cb4ae","Type":"ContainerDied","Data":"f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0"} Nov 03 19:32:27.666765 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.666598 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9" event={"ID":"ddca4421-6d98-415e-9848-80ec181cb4ae","Type":"ContainerDied","Data":"17a2876f300d1a1a51b0ca31dbfe0f0c6b5b24898f5f83c0e5bb76c775d49dab"} Nov 03 19:32:27.666765 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.666628 5019 scope.go:122] "RemoveContainer" containerID="f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0" Nov 03 19:32:27.678605 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.678580 5019 scope.go:122] "RemoveContainer" containerID="5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99" Nov 03 19:32:27.687564 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.687543 5019 scope.go:122] "RemoveContainer" containerID="f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0" Nov 03 19:32:27.687833 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:32:27.687815 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0\": container with ID starting with f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0 not found: ID does not exist" containerID="f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0" Nov 03 19:32:27.687907 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.687841 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0"} err="failed to get container status \"f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0\": rpc error: code = NotFound desc = could not find container \"f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0\": container with ID starting with f810b915a7844b91758ed7845bc36a6e0f8f7b3341419d476385463464a195a0 not found: ID does not exist" Nov 03 19:32:27.687907 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.687858 5019 scope.go:122] "RemoveContainer" containerID="5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99" Nov 03 19:32:27.688106 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:32:27.688086 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99\": container with ID starting with 5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99 not found: ID does not exist" containerID="5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99" Nov 03 19:32:27.688165 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.688113 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99"} err="failed to get container status \"5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99\": rpc error: code = NotFound desc = could not find container \"5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99\": container with ID starting with 5519a09460b652b3b09e28573277fc4477967f8ccb345fcf3066cbc1f380cb99 not found: ID does not exist" Nov 03 19:32:27.822474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.822436 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9"] Nov 03 19:32:27.868991 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:27.868956 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-h7fh9"] Nov 03 19:32:28.151076 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:28.151044 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" path="/var/lib/kubelet/pods/ddca4421-6d98-415e-9848-80ec181cb4ae/volumes" Nov 03 19:32:30.096264 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:30.096224 5019 scope.go:122] "RemoveContainer" containerID="447f5a726744a69bc2837d6f00d08164115bdb24a3fa0ad4dd2f0803c1ac348b" Nov 03 19:32:36.620274 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:36.620239 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x"] Nov 03 19:32:36.721709 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:36.721678 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4"] Nov 03 19:32:41.296056 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:32:41.296021 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:32:41.621589 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:41.621498 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" podUID="7ae27eaa-ff30-4e3f-b671-5c5d429f9094" containerName="payload-processing" containerID="cri-o://37e5e03739d3d549d16cafa36382e601accbde4d31ecd9384932d6b9e0de82b4" gracePeriod=25 Nov 03 19:32:41.722088 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:41.722031 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" podUID="d6febc76-f9af-48b5-ac02-1f7805eb1d0a" containerName="payload-pre-processing" containerID="cri-o://0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91" gracePeriod=25 Nov 03 19:32:41.784823 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:41.784787 5019 generic.go:356] "Generic (PLEG): container finished" podID="7ae27eaa-ff30-4e3f-b671-5c5d429f9094" containerID="37e5e03739d3d549d16cafa36382e601accbde4d31ecd9384932d6b9e0de82b4" exitCode=0 Nov 03 19:32:41.784970 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:41.784862 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" event={"ID":"7ae27eaa-ff30-4e3f-b671-5c5d429f9094","Type":"ContainerDied","Data":"37e5e03739d3d549d16cafa36382e601accbde4d31ecd9384932d6b9e0de82b4"} Nov 03 19:32:41.917532 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:41.917500 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" Nov 03 19:32:41.988281 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:41.988258 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" Nov 03 19:32:42.034112 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.034087 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-plugins-config-volume\") pod \"7ae27eaa-ff30-4e3f-b671-5c5d429f9094\" (UID: \"7ae27eaa-ff30-4e3f-b671-5c5d429f9094\") " Nov 03 19:32:42.034247 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.034223 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-kube-api-access-t9lqq\" (UniqueName: \"kubernetes.io/projected/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-kube-api-access-t9lqq\") pod \"7ae27eaa-ff30-4e3f-b671-5c5d429f9094\" (UID: \"7ae27eaa-ff30-4e3f-b671-5c5d429f9094\") " Nov 03 19:32:42.034483 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.034461 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-plugins-config-volume" pod "7ae27eaa-ff30-4e3f-b671-5c5d429f9094" (UID: "7ae27eaa-ff30-4e3f-b671-5c5d429f9094"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:42.034727 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.034712 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-plugins-config-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:42.036657 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.036634 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-kube-api-access-t9lqq" pod "7ae27eaa-ff30-4e3f-b671-5c5d429f9094" (UID: "7ae27eaa-ff30-4e3f-b671-5c5d429f9094"). InnerVolumeSpecName "kube-api-access-t9lqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:42.135602 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.135533 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-plugins-config-volume\") pod \"d6febc76-f9af-48b5-ac02-1f7805eb1d0a\" (UID: \"d6febc76-f9af-48b5-ac02-1f7805eb1d0a\") " Nov 03 19:32:42.135742 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.135607 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-kube-api-access-gmpdc\" (UniqueName: \"kubernetes.io/projected/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-kube-api-access-gmpdc\") pod \"d6febc76-f9af-48b5-ac02-1f7805eb1d0a\" (UID: \"d6febc76-f9af-48b5-ac02-1f7805eb1d0a\") " Nov 03 19:32:42.135911 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.135889 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-plugins-config-volume" pod "d6febc76-f9af-48b5-ac02-1f7805eb1d0a" (UID: "d6febc76-f9af-48b5-ac02-1f7805eb1d0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:42.135993 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.135980 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9lqq\" (UniqueName: \"kubernetes.io/projected/7ae27eaa-ff30-4e3f-b671-5c5d429f9094-kube-api-access-t9lqq\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:42.137821 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.137795 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-kube-api-access-gmpdc" pod "d6febc76-f9af-48b5-ac02-1f7805eb1d0a" (UID: "d6febc76-f9af-48b5-ac02-1f7805eb1d0a"). InnerVolumeSpecName "kube-api-access-gmpdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:42.236956 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.236927 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmpdc\" (UniqueName: \"kubernetes.io/projected/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-kube-api-access-gmpdc\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:42.236956 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.236953 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6febc76-f9af-48b5-ac02-1f7805eb1d0a-plugins-config-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:42.792055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.792018 5019 generic.go:356] "Generic (PLEG): container finished" podID="d6febc76-f9af-48b5-ac02-1f7805eb1d0a" containerID="0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91" exitCode=0 Nov 03 19:32:42.792618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.792067 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" Nov 03 19:32:42.792618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.792098 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" event={"ID":"d6febc76-f9af-48b5-ac02-1f7805eb1d0a","Type":"ContainerDied","Data":"0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91"} Nov 03 19:32:42.792618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.792161 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4" event={"ID":"d6febc76-f9af-48b5-ac02-1f7805eb1d0a","Type":"ContainerDied","Data":"b1eba7116c37b5f2d268e246beaa5a18ec46dd6c676f7ed9f30da1d180bd7388"} Nov 03 19:32:42.792618 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.792192 5019 scope.go:122] "RemoveContainer" containerID="0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91" Nov 03 19:32:42.793771 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.793755 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" Nov 03 19:32:42.793861 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.793778 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x" event={"ID":"7ae27eaa-ff30-4e3f-b671-5c5d429f9094","Type":"ContainerDied","Data":"5a14a0c2c6deb993a7ecac50f935af610c630838459e2057dffff632a7bcb850"} Nov 03 19:32:42.803190 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.803113 5019 scope.go:122] "RemoveContainer" containerID="0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91" Nov 03 19:32:42.803480 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:32:42.803461 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91\": container with ID starting with 0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91 not found: ID does not exist" containerID="0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91" Nov 03 19:32:42.803540 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.803495 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91"} err="failed to get container status \"0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91\": rpc error: code = NotFound desc = could not find container \"0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91\": container with ID starting with 0a4df1c74ca28f9c1926cf24f505beb9a1ad1febb0ef50517c83e3f4c5247e91 not found: ID does not exist" Nov 03 19:32:42.803540 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.803512 5019 scope.go:122] "RemoveContainer" containerID="37e5e03739d3d549d16cafa36382e601accbde4d31ecd9384932d6b9e0de82b4" Nov 03 19:32:42.992503 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.992474 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x"] Nov 03 19:32:42.996164 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:42.996138 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9dcd6c-79d7484f9b-2n24x"] Nov 03 19:32:43.098191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:43.098161 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4"] Nov 03 19:32:43.104038 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:43.104002 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9dcd6c-5b68f8c7ff-mnbb4"] Nov 03 19:32:44.151070 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:44.151040 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae27eaa-ff30-4e3f-b671-5c5d429f9094" path="/var/lib/kubelet/pods/7ae27eaa-ff30-4e3f-b671-5c5d429f9094/volumes" Nov 03 19:32:44.151518 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:44.151504 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6febc76-f9af-48b5-ac02-1f7805eb1d0a" path="/var/lib/kubelet/pods/d6febc76-f9af-48b5-ac02-1f7805eb1d0a/volumes" Nov 03 19:32:52.427588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.427545 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj"] Nov 03 19:32:52.428223 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428203 5019 cpu_manager.go:414] "containerMap: removing container" podUID="11d500e8-f34b-4fe8-ad68-363552ebd6f6" containerName="istio-proxy" Nov 03 19:32:52.428278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428225 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d500e8-f34b-4fe8-ad68-363552ebd6f6" containerName="istio-proxy" Nov 03 19:32:52.428278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428245 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" Nov 03 19:32:52.428278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428250 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" Nov 03 19:32:52.428278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428260 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d6febc76-f9af-48b5-ac02-1f7805eb1d0a" containerName="payload-pre-processing" Nov 03 19:32:52.428278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428265 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6febc76-f9af-48b5-ac02-1f7805eb1d0a" containerName="payload-pre-processing" Nov 03 19:32:52.428278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428276 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7ae27eaa-ff30-4e3f-b671-5c5d429f9094" containerName="payload-processing" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428281 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae27eaa-ff30-4e3f-b671-5c5d429f9094" containerName="payload-processing" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428292 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf" containerName="maas-api" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428299 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf" containerName="maas-api" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428344 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b54643ca-250e-49db-b008-f7f6e15aca04" containerName="maas-api" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428349 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b54643ca-250e-49db-b008-f7f6e15aca04" containerName="maas-api" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428361 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f4773c82-6fb1-4108-aa54-a6e5635a847b" containerName="maas-api" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428368 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4773c82-6fb1-4108-aa54-a6e5635a847b" containerName="maas-api" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428464 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4773c82-6fb1-4108-aa54-a6e5635a847b" containerName="maas-api" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428477 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d500e8-f34b-4fe8-ad68-363552ebd6f6" containerName="istio-proxy" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428485 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428490 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6febc76-f9af-48b5-ac02-1f7805eb1d0a" containerName="payload-pre-processing" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428497 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae27eaa-ff30-4e3f-b671-5c5d429f9094" containerName="payload-processing" Nov 03 19:32:52.428501 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428503 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a579a5f2-a4c4-4152-b71a-b4a35fc8ccdf" containerName="maas-api" Nov 03 19:32:52.428982 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428511 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b59638-64eb-4da2-93aa-aa8f37354323" containerName="istio-proxy" Nov 03 19:32:52.428982 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428518 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" Nov 03 19:32:52.428982 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.428525 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b54643ca-250e-49db-b008-f7f6e15aca04" containerName="maas-api" Nov 03 19:32:52.433361 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.433322 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:52.454797 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.454769 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-maas-api-tls\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:52.454915 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.454823 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-metrics-certs\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:52.454915 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.454871 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk2pq\" (UniqueName: \"kubernetes.io/projected/d2bbf06c-ae28-42e0-8767-08283df56709-kube-api-access-vk2pq\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:52.455022 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.455008 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2bbf06c-ae28-42e0-8767-08283df56709-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:52.482191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.482169 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-24cdad-serving-cert\"" Nov 03 19:32:52.531473 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.531439 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj"] Nov 03 19:32:52.555903 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.555871 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-metrics-certs\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:52.556076 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.555920 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk2pq\" (UniqueName: \"kubernetes.io/projected/d2bbf06c-ae28-42e0-8767-08283df56709-kube-api-access-vk2pq\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:52.556076 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.556037 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2bbf06c-ae28-42e0-8767-08283df56709-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:52.556198 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.556107 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-maas-api-tls\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:52.556262 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:32:52.556238 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-serving-cert: secret "maas-api-e2e-inf-b-24cdad-serving-cert" not found Nov 03 19:32:52.556364 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:32:52.556309 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-maas-api-tls podName:d2bbf06c-ae28-42e0-8767-08283df56709 nodeName:}" failed. No retries permitted until 2026-11-03 19:32:53.056289061 +0000 UTC m=+2907.028850239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-maas-api-tls") pod "maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" (UID: "d2bbf06c-ae28-42e0-8767-08283df56709") : secret "maas-api-e2e-inf-b-24cdad-serving-cert" not found Nov 03 19:32:52.557121 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.557085 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2bbf06c-ae28-42e0-8767-08283df56709-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:52.558649 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.558622 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-metrics-certs\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:52.614045 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:52.614013 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk2pq\" (UniqueName: \"kubernetes.io/projected/d2bbf06c-ae28-42e0-8767-08283df56709-kube-api-access-vk2pq\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:53.083559 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:53.083521 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-maas-api-tls\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:53.086083 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:53.086054 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-maas-api-tls\") pod \"maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:53.347235 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:53.347152 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:53.721721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:53.721695 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj"] Nov 03 19:32:53.722798 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:32:53.722759 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bbf06c_ae28_42e0_8767_08283df56709.slice/crio-dde1a3d804f7dc1787147c2e8c31c726c7a91d0a054fcd4a608789c2b415d55f WatchSource:0}: Error finding container dde1a3d804f7dc1787147c2e8c31c726c7a91d0a054fcd4a608789c2b415d55f: Status 404 returned error can't find the container with id dde1a3d804f7dc1787147c2e8c31c726c7a91d0a054fcd4a608789c2b415d55f Nov 03 19:32:53.724495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:53.724474 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:32:53.865656 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:53.865623 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" event={"ID":"d2bbf06c-ae28-42e0-8767-08283df56709","Type":"ContainerStarted","Data":"dde1a3d804f7dc1787147c2e8c31c726c7a91d0a054fcd4a608789c2b415d55f"} Nov 03 19:32:54.873256 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:54.873169 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" event={"ID":"d2bbf06c-ae28-42e0-8767-08283df56709","Type":"ContainerStarted","Data":"40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac"} Nov 03 19:32:54.873256 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:54.873238 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:32:54.985253 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:32:54.985201 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" podStartSLOduration=2.156481474 podStartE2EDuration="2.985187443s" totalImagesPullingTime="828.705969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:54.982925296 +0000 UTC m=+2908.955486478" watchObservedRunningTime="2026-11-03 19:32:54.985187443 +0000 UTC m=+2908.957748625" Nov 03 19:33:00.885558 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:00.885527 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:33:01.190988 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.190953 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw"] Nov 03 19:33:01.191192 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.191167 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" podUID="4d5e2740-ab79-4f6e-97cb-bac3339fde21" containerName="maas-api" containerID="cri-o://11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29" gracePeriod=30 Nov 03 19:33:01.492053 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.492028 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 19:33:01.586350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.584461 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d5e2740-ab79-4f6e-97cb-bac3339fde21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5e2740-ab79-4f6e-97cb-bac3339fde21-openshift-service-ca\") pod \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " Nov 03 19:33:01.586350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.584648 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-maas-api-tls\") pod \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " Nov 03 19:33:01.586350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.584784 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5e2740-ab79-4f6e-97cb-bac3339fde21-kube-api-access-dv84d\" (UniqueName: \"kubernetes.io/projected/4d5e2740-ab79-4f6e-97cb-bac3339fde21-kube-api-access-dv84d\") pod \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " Nov 03 19:33:01.586350 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.584814 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-metrics-certs\") pod \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\" (UID: \"4d5e2740-ab79-4f6e-97cb-bac3339fde21\") " Nov 03 19:33:01.586736 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.586646 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5e2740-ab79-4f6e-97cb-bac3339fde21-openshift-service-ca" pod "4d5e2740-ab79-4f6e-97cb-bac3339fde21" (UID: "4d5e2740-ab79-4f6e-97cb-bac3339fde21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:01.596122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.593068 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-metrics-certs" pod "4d5e2740-ab79-4f6e-97cb-bac3339fde21" (UID: "4d5e2740-ab79-4f6e-97cb-bac3339fde21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:33:01.596122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.595683 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5e2740-ab79-4f6e-97cb-bac3339fde21-kube-api-access-dv84d" pod "4d5e2740-ab79-4f6e-97cb-bac3339fde21" (UID: "4d5e2740-ab79-4f6e-97cb-bac3339fde21"). InnerVolumeSpecName "kube-api-access-dv84d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:01.597091 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.596862 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-maas-api-tls" pod "4d5e2740-ab79-4f6e-97cb-bac3339fde21" (UID: "4d5e2740-ab79-4f6e-97cb-bac3339fde21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:33:01.686556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.686522 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv84d\" (UniqueName: \"kubernetes.io/projected/4d5e2740-ab79-4f6e-97cb-bac3339fde21-kube-api-access-dv84d\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:01.686556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.686551 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:01.686556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.686562 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5e2740-ab79-4f6e-97cb-bac3339fde21-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:01.686782 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.686571 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5e2740-ab79-4f6e-97cb-bac3339fde21-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:01.924211 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.924175 5019 generic.go:356] "Generic (PLEG): container finished" podID="4d5e2740-ab79-4f6e-97cb-bac3339fde21" containerID="11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29" exitCode=0 Nov 03 19:33:01.924647 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.924245 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" Nov 03 19:33:01.924647 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.924259 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" event={"ID":"4d5e2740-ab79-4f6e-97cb-bac3339fde21","Type":"ContainerDied","Data":"11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29"} Nov 03 19:33:01.924647 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.924368 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw" event={"ID":"4d5e2740-ab79-4f6e-97cb-bac3339fde21","Type":"ContainerDied","Data":"89bae917f1f51fa48532c50dd53abad4b2e5e43790eea5bff94c1eb6e9e2a4a8"} Nov 03 19:33:01.924647 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.924394 5019 scope.go:122] "RemoveContainer" containerID="11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29" Nov 03 19:33:01.934903 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.934883 5019 scope.go:122] "RemoveContainer" containerID="11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29" Nov 03 19:33:01.935186 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:33:01.935165 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29\": container with ID starting with 11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29 not found: ID does not exist" containerID="11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29" Nov 03 19:33:01.935253 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:01.935194 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29"} err="failed to get container status \"11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29\": rpc error: code = NotFound desc = could not find container \"11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29\": container with ID starting with 11b339a924d67621b77bf0797487bd27501268a26198e4e43c3d89759f76ef29 not found: ID does not exist" Nov 03 19:33:02.082499 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:02.082456 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw"] Nov 03 19:33:02.137352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:02.135581 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-010ed3-64c54959d4-fw6fw"] Nov 03 19:33:02.161349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:02.160549 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5e2740-ab79-4f6e-97cb-bac3339fde21" path="/var/lib/kubelet/pods/4d5e2740-ab79-4f6e-97cb-bac3339fde21/volumes" Nov 03 19:33:06.936525 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:06.936449 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4"] Nov 03 19:33:06.936973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:06.936647 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" podUID="81073a4c-136f-485e-91ec-329f36057350" containerName="maas-api" containerID="cri-o://e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e" gracePeriod=30 Nov 03 19:33:07.004593 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:33:07.004547 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81073a4c_136f_485e_91ec_329f36057350.slice/crio-e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81073a4c_136f_485e_91ec_329f36057350.slice/crio-conmon-e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e.scope\": RecentStats: unable to find data in memory cache]" Nov 03 19:33:07.239976 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.239951 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:33:07.291165 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.291130 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81073a4c-136f-485e-91ec-329f36057350-kube-api-access-6vj7k\" (UniqueName: \"kubernetes.io/projected/81073a4c-136f-485e-91ec-329f36057350-kube-api-access-6vj7k\") pod \"81073a4c-136f-485e-91ec-329f36057350\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " Nov 03 19:33:07.291367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.291249 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-metrics-certs\" (UniqueName: \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-metrics-certs\") pod \"81073a4c-136f-485e-91ec-329f36057350\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " Nov 03 19:33:07.291367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.291317 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-maas-api-tls\") pod \"81073a4c-136f-485e-91ec-329f36057350\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " Nov 03 19:33:07.291499 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.291386 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81073a4c-136f-485e-91ec-329f36057350-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81073a4c-136f-485e-91ec-329f36057350-openshift-service-ca\") pod \"81073a4c-136f-485e-91ec-329f36057350\" (UID: \"81073a4c-136f-485e-91ec-329f36057350\") " Nov 03 19:33:07.291866 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.291837 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81073a4c-136f-485e-91ec-329f36057350-openshift-service-ca" pod "81073a4c-136f-485e-91ec-329f36057350" (UID: "81073a4c-136f-485e-91ec-329f36057350"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:07.291982 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.291933 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81073a4c-136f-485e-91ec-329f36057350-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:07.293529 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.293493 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81073a4c-136f-485e-91ec-329f36057350-kube-api-access-6vj7k" pod "81073a4c-136f-485e-91ec-329f36057350" (UID: "81073a4c-136f-485e-91ec-329f36057350"). InnerVolumeSpecName "kube-api-access-6vj7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:07.293628 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.293604 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-metrics-certs" pod "81073a4c-136f-485e-91ec-329f36057350" (UID: "81073a4c-136f-485e-91ec-329f36057350"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:33:07.293906 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.293887 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-maas-api-tls" pod "81073a4c-136f-485e-91ec-329f36057350" (UID: "81073a4c-136f-485e-91ec-329f36057350"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:33:07.393417 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.393373 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:07.393417 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.393415 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81073a4c-136f-485e-91ec-329f36057350-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:07.393619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.393431 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vj7k\" (UniqueName: \"kubernetes.io/projected/81073a4c-136f-485e-91ec-329f36057350-kube-api-access-6vj7k\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:07.965175 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.965139 5019 generic.go:356] "Generic (PLEG): container finished" podID="81073a4c-136f-485e-91ec-329f36057350" containerID="e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e" exitCode=0 Nov 03 19:33:07.965717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.965193 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" Nov 03 19:33:07.965717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.965213 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" event={"ID":"81073a4c-136f-485e-91ec-329f36057350","Type":"ContainerDied","Data":"e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e"} Nov 03 19:33:07.965717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.965282 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4" event={"ID":"81073a4c-136f-485e-91ec-329f36057350","Type":"ContainerDied","Data":"03d106113b3931d3a94a5b0414f8f415ebd7682614b3316268364dc181582444"} Nov 03 19:33:07.965717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.965311 5019 scope.go:122] "RemoveContainer" containerID="e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e" Nov 03 19:33:07.976298 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.976281 5019 scope.go:122] "RemoveContainer" containerID="e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e" Nov 03 19:33:07.976648 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:33:07.976629 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e\": container with ID starting with e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e not found: ID does not exist" containerID="e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e" Nov 03 19:33:07.976721 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:07.976659 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e"} err="failed to get container status \"e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e\": rpc error: code = NotFound desc = could not find container \"e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e\": container with ID starting with e5a52f5f33d42b55505aaf0a9fd73c51a2f572958a967dc894912386596a701e not found: ID does not exist" Nov 03 19:33:08.141521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:08.141489 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4"] Nov 03 19:33:08.152621 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:08.152588 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-73a216-79fb8f4d9f-kbzx4"] Nov 03 19:33:10.150808 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:10.150772 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81073a4c-136f-485e-91ec-329f36057350" path="/var/lib/kubelet/pods/81073a4c-136f-485e-91ec-329f36057350/volumes" Nov 03 19:33:11.295703 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:33:11.295667 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:33:13.096254 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.096222 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4"] Nov 03 19:33:13.096742 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.096491 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" podUID="b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b" containerName="maas-api" containerID="cri-o://9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b" gracePeriod=30 Nov 03 19:33:13.414886 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.414861 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:33:13.519065 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.519031 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-maas-api-tls\") pod \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " Nov 03 19:33:13.519275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.519094 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-kube-api-access-llg89\" (UniqueName: \"kubernetes.io/projected/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-kube-api-access-llg89\") pod \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " Nov 03 19:33:13.519275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.519128 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-openshift-service-ca\") pod \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " Nov 03 19:33:13.519275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.519175 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-metrics-certs\") pod \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\" (UID: \"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b\") " Nov 03 19:33:13.519543 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.519494 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-openshift-service-ca" pod "b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b" (UID: "b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:13.519661 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.519641 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:13.521440 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.521406 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-maas-api-tls" pod "b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b" (UID: "b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:33:13.521554 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.521432 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-kube-api-access-llg89" pod "b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b" (UID: "b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b"). InnerVolumeSpecName "kube-api-access-llg89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:13.521670 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.521651 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-metrics-certs" pod "b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b" (UID: "b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:33:13.621084 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.621054 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llg89\" (UniqueName: \"kubernetes.io/projected/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-kube-api-access-llg89\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:13.621084 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.621084 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:13.621371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:13.621093 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:14.029553 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:14.029518 5019 generic.go:356] "Generic (PLEG): container finished" podID="b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b" containerID="9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b" exitCode=0 Nov 03 19:33:14.029865 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:14.029600 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" Nov 03 19:33:14.030054 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:14.029623 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" event={"ID":"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b","Type":"ContainerDied","Data":"9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b"} Nov 03 19:33:14.030195 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:14.030178 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4" event={"ID":"b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b","Type":"ContainerDied","Data":"b6ea6bbab12d4bc184c706127ff06334d73602759cba9983b500c735c6dd642d"} Nov 03 19:33:14.030313 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:14.030300 5019 scope.go:122] "RemoveContainer" containerID="9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b" Nov 03 19:33:14.068589 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:14.065736 5019 scope.go:122] "RemoveContainer" containerID="9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b" Nov 03 19:33:14.068589 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:33:14.066176 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b\": container with ID starting with 9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b not found: ID does not exist" containerID="9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b" Nov 03 19:33:14.068589 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:14.066217 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b"} err="failed to get container status \"9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b\": rpc error: code = NotFound desc = could not find container \"9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b\": container with ID starting with 9d82cab3e19d92c8720ad619c4d319ba50ecd5da6e4ec26938d4fcef300fca7b not found: ID does not exist" Nov 03 19:33:14.184555 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:14.184517 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4"] Nov 03 19:33:14.230808 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:14.230775 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9dcd6c-6b997666c-q6xh4"] Nov 03 19:33:16.046462 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:16.046433 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv"] Nov 03 19:33:16.047119 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:16.046658 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" containerID="cri-o://ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc" gracePeriod=30 Nov 03 19:33:16.149874 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:16.149843 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b" path="/var/lib/kubelet/pods/b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b/volumes" Nov 03 19:33:21.318788 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.318733 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:33:21.413821 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.413786 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-credential-socket\") pod \"501946c2-6e3d-4fb2-9123-e105014fb260\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " Nov 03 19:33:21.414005 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.413881 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-kube-api-access-khpvh\" (UniqueName: \"kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-kube-api-access-khpvh\") pod \"501946c2-6e3d-4fb2-9123-e105014fb260\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " Nov 03 19:33:21.414005 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.413914 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-socket\") pod \"501946c2-6e3d-4fb2-9123-e105014fb260\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " Nov 03 19:33:21.414005 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.413948 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-certs\") pod \"501946c2-6e3d-4fb2-9123-e105014fb260\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " Nov 03 19:33:21.414158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414006 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/501946c2-6e3d-4fb2-9123-e105014fb260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501946c2-6e3d-4fb2-9123-e105014fb260-istio-podinfo\") pod \"501946c2-6e3d-4fb2-9123-e105014fb260\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " Nov 03 19:33:21.414158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414055 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-istio-token\" (UniqueName: \"kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-istio-token\") pod \"501946c2-6e3d-4fb2-9123-e105014fb260\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " Nov 03 19:33:21.414158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414065 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-credential-socket" pod "501946c2-6e3d-4fb2-9123-e105014fb260" (UID: "501946c2-6e3d-4fb2-9123-e105014fb260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:21.414158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414147 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-data\") pod \"501946c2-6e3d-4fb2-9123-e105014fb260\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " Nov 03 19:33:21.414381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414235 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-socket" pod "501946c2-6e3d-4fb2-9123-e105014fb260" (UID: "501946c2-6e3d-4fb2-9123-e105014fb260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:21.414381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414245 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-envoy\") pod \"501946c2-6e3d-4fb2-9123-e105014fb260\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " Nov 03 19:33:21.414381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414319 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/501946c2-6e3d-4fb2-9123-e105014fb260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501946c2-6e3d-4fb2-9123-e105014fb260-istiod-ca-cert\") pod \"501946c2-6e3d-4fb2-9123-e105014fb260\" (UID: \"501946c2-6e3d-4fb2-9123-e105014fb260\") " Nov 03 19:33:21.414598 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414573 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-data" pod "501946c2-6e3d-4fb2-9123-e105014fb260" (UID: "501946c2-6e3d-4fb2-9123-e105014fb260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:21.414671 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414640 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-certs" pod "501946c2-6e3d-4fb2-9123-e105014fb260" (UID: "501946c2-6e3d-4fb2-9123-e105014fb260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:21.414788 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414762 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501946c2-6e3d-4fb2-9123-e105014fb260-istiod-ca-cert" pod "501946c2-6e3d-4fb2-9123-e105014fb260" (UID: "501946c2-6e3d-4fb2-9123-e105014fb260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:21.414959 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414936 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:21.415087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414962 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501946c2-6e3d-4fb2-9123-e105014fb260-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:21.415087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414977 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:21.415087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.414991 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:21.415087 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.415005 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:21.416353 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.416300 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-kube-api-access-khpvh" pod "501946c2-6e3d-4fb2-9123-e105014fb260" (UID: "501946c2-6e3d-4fb2-9123-e105014fb260"). InnerVolumeSpecName "kube-api-access-khpvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:21.416449 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.416430 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-istio-token" pod "501946c2-6e3d-4fb2-9123-e105014fb260" (UID: "501946c2-6e3d-4fb2-9123-e105014fb260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:21.416538 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.416517 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/501946c2-6e3d-4fb2-9123-e105014fb260-istio-podinfo" pod "501946c2-6e3d-4fb2-9123-e105014fb260" (UID: "501946c2-6e3d-4fb2-9123-e105014fb260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:33:21.416676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.416659 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-envoy" pod "501946c2-6e3d-4fb2-9123-e105014fb260" (UID: "501946c2-6e3d-4fb2-9123-e105014fb260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:21.516627 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.516589 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501946c2-6e3d-4fb2-9123-e105014fb260-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:21.516627 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.516629 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khpvh\" (UniqueName: \"kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-kube-api-access-khpvh\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:21.516828 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.516644 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501946c2-6e3d-4fb2-9123-e105014fb260-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:21.516828 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.516658 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501946c2-6e3d-4fb2-9123-e105014fb260-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:21.972758 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.972721 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs"] Nov 03 19:33:21.973004 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:21.972976 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" podUID="641da692-25ae-4e05-9e06-2417ab73918d" containerName="istio-proxy" containerID="cri-o://1778f4fc9111809980b7cefada33d69d25ff46b80ae79cfef3ebd8bae41ce580" gracePeriod=30 Nov 03 19:33:22.083820 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:22.083789 5019 generic.go:356] "Generic (PLEG): container finished" podID="501946c2-6e3d-4fb2-9123-e105014fb260" containerID="ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc" exitCode=0 Nov 03 19:33:22.084039 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:22.083856 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" Nov 03 19:33:22.084039 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:22.083879 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" event={"ID":"501946c2-6e3d-4fb2-9123-e105014fb260","Type":"ContainerDied","Data":"ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc"} Nov 03 19:33:22.084039 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:22.083948 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv" event={"ID":"501946c2-6e3d-4fb2-9123-e105014fb260","Type":"ContainerDied","Data":"b7cd2bb022c63354d0fa7b8ec7b276bd0fab11d63614f7a80defe62107a0fd52"} Nov 03 19:33:22.084039 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:22.083978 5019 scope.go:122] "RemoveContainer" containerID="ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc" Nov 03 19:33:22.108448 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:22.108420 5019 scope.go:122] "RemoveContainer" containerID="c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64" Nov 03 19:33:22.131141 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:22.131068 5019 scope.go:122] "RemoveContainer" containerID="ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc" Nov 03 19:33:22.132258 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:33:22.132226 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc\": container with ID starting with ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc not found: ID does not exist" containerID="ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc" Nov 03 19:33:22.132507 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:22.132476 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc"} err="failed to get container status \"ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc\": rpc error: code = NotFound desc = could not find container \"ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc\": container with ID starting with ca767ebbf83e78964c7cc6efa7464f9100fcd167c95d9d2854d1cfd0a8d55edc not found: ID does not exist" Nov 03 19:33:22.132648 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:22.132635 5019 scope.go:122] "RemoveContainer" containerID="c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64" Nov 03 19:33:22.133060 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:33:22.133036 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64\": container with ID starting with c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64 not found: ID does not exist" containerID="c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64" Nov 03 19:33:22.134656 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:22.134617 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64"} err="failed to get container status \"c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64\": rpc error: code = NotFound desc = could not find container \"c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64\": container with ID starting with c614486317407c19ec623b51900ad591f665a6c2d4f0f42df37373ae03eedc64 not found: ID does not exist" Nov 03 19:33:22.252626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:22.252549 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv"] Nov 03 19:33:22.256866 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:22.256839 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-73a216-openshift-default-bcb57fcf9-hbblv"] Nov 03 19:33:24.150697 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:24.150665 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" path="/var/lib/kubelet/pods/501946c2-6e3d-4fb2-9123-e105014fb260/volumes" Nov 03 19:33:27.115782 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.115750 5019 generic.go:356] "Generic (PLEG): container finished" podID="641da692-25ae-4e05-9e06-2417ab73918d" containerID="1778f4fc9111809980b7cefada33d69d25ff46b80ae79cfef3ebd8bae41ce580" exitCode=0 Nov 03 19:33:27.116170 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.115817 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" event={"ID":"641da692-25ae-4e05-9e06-2417ab73918d","Type":"ContainerDied","Data":"1778f4fc9111809980b7cefada33d69d25ff46b80ae79cfef3ebd8bae41ce580"} Nov 03 19:33:27.116170 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.115882 5019 scope.go:122] "RemoveContainer" containerID="e59c1a0755e146175e8717f9766142ea60347220ffec2358f039f5b6c323445f" Nov 03 19:33:27.257314 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.257255 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:33:27.286456 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.286425 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/641da692-25ae-4e05-9e06-2417ab73918d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641da692-25ae-4e05-9e06-2417ab73918d-istio-podinfo\") pod \"641da692-25ae-4e05-9e06-2417ab73918d\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " Nov 03 19:33:27.286605 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.286535 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-envoy\") pod \"641da692-25ae-4e05-9e06-2417ab73918d\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " Nov 03 19:33:27.286605 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.286575 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641da692-25ae-4e05-9e06-2417ab73918d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641da692-25ae-4e05-9e06-2417ab73918d-istiod-ca-cert\") pod \"641da692-25ae-4e05-9e06-2417ab73918d\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " Nov 03 19:33:27.286682 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.286633 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-credential-socket\") pod \"641da692-25ae-4e05-9e06-2417ab73918d\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " Nov 03 19:33:27.286726 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.286704 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-istio-token\" (UniqueName: \"kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-istio-token\") pod \"641da692-25ae-4e05-9e06-2417ab73918d\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " Nov 03 19:33:27.286785 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.286752 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-certs\") pod \"641da692-25ae-4e05-9e06-2417ab73918d\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " Nov 03 19:33:27.286840 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.286782 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-data\") pod \"641da692-25ae-4e05-9e06-2417ab73918d\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " Nov 03 19:33:27.286840 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.286817 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-kube-api-access-tdn72\" (UniqueName: \"kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-kube-api-access-tdn72\") pod \"641da692-25ae-4e05-9e06-2417ab73918d\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " Nov 03 19:33:27.286967 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.286952 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-socket\") pod \"641da692-25ae-4e05-9e06-2417ab73918d\" (UID: \"641da692-25ae-4e05-9e06-2417ab73918d\") " Nov 03 19:33:27.287135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.287026 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641da692-25ae-4e05-9e06-2417ab73918d-istiod-ca-cert" pod "641da692-25ae-4e05-9e06-2417ab73918d" (UID: "641da692-25ae-4e05-9e06-2417ab73918d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:27.287209 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.287129 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-certs" pod "641da692-25ae-4e05-9e06-2417ab73918d" (UID: "641da692-25ae-4e05-9e06-2417ab73918d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:27.287413 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.287396 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641da692-25ae-4e05-9e06-2417ab73918d-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:27.287486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.287417 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:27.287753 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.287713 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-data" pod "641da692-25ae-4e05-9e06-2417ab73918d" (UID: "641da692-25ae-4e05-9e06-2417ab73918d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:27.287753 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.287727 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-socket" pod "641da692-25ae-4e05-9e06-2417ab73918d" (UID: "641da692-25ae-4e05-9e06-2417ab73918d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:27.287920 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.287832 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-credential-socket" pod "641da692-25ae-4e05-9e06-2417ab73918d" (UID: "641da692-25ae-4e05-9e06-2417ab73918d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:27.289176 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.289151 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-envoy" pod "641da692-25ae-4e05-9e06-2417ab73918d" (UID: "641da692-25ae-4e05-9e06-2417ab73918d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:27.289273 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.289150 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/641da692-25ae-4e05-9e06-2417ab73918d-istio-podinfo" pod "641da692-25ae-4e05-9e06-2417ab73918d" (UID: "641da692-25ae-4e05-9e06-2417ab73918d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:33:27.289532 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.289506 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-istio-token" pod "641da692-25ae-4e05-9e06-2417ab73918d" (UID: "641da692-25ae-4e05-9e06-2417ab73918d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:27.291156 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.291138 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-kube-api-access-tdn72" pod "641da692-25ae-4e05-9e06-2417ab73918d" (UID: "641da692-25ae-4e05-9e06-2417ab73918d"). InnerVolumeSpecName "kube-api-access-tdn72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:27.388633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.388517 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:27.388633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.388548 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:27.388633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.388563 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdn72\" (UniqueName: \"kubernetes.io/projected/641da692-25ae-4e05-9e06-2417ab73918d-kube-api-access-tdn72\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:27.388633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.388578 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:27.388633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.388590 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641da692-25ae-4e05-9e06-2417ab73918d-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:27.388633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.388603 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:27.388633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:27.388616 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641da692-25ae-4e05-9e06-2417ab73918d-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:28.123143 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:28.123112 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" event={"ID":"641da692-25ae-4e05-9e06-2417ab73918d","Type":"ContainerDied","Data":"b3f47afb84795f0bf20e525d797e7c9c37b79dc0fb69db09c470b006d3dab6fd"} Nov 03 19:33:28.123654 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:28.123164 5019 scope.go:122] "RemoveContainer" containerID="1778f4fc9111809980b7cefada33d69d25ff46b80ae79cfef3ebd8bae41ce580" Nov 03 19:33:28.123654 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:28.123120 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs" Nov 03 19:33:28.285643 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:28.284928 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs"] Nov 03 19:33:28.300635 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:28.300604 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9dcd6c-openshift-default-5bc7546d4c-8gnvs"] Nov 03 19:33:30.150599 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:30.150572 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641da692-25ae-4e05-9e06-2417ab73918d" path="/var/lib/kubelet/pods/641da692-25ae-4e05-9e06-2417ab73918d/volumes" Nov 03 19:33:41.301100 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:33:41.301062 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:33:52.869588 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:52.869551 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2"] Nov 03 19:33:52.870216 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:52.869825 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" podUID="3dbb737f-ccec-4959-af11-33486428f42f" containerName="maas-api" containerID="cri-o://2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e" gracePeriod=30 Nov 03 19:33:53.154831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.154809 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:33:53.289144 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.289106 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbb737f-ccec-4959-af11-33486428f42f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbb737f-ccec-4959-af11-33486428f42f-openshift-service-ca\") pod \"3dbb737f-ccec-4959-af11-33486428f42f\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " Nov 03 19:33:53.289390 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.289168 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-metrics-certs\") pod \"3dbb737f-ccec-4959-af11-33486428f42f\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " Nov 03 19:33:53.289390 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.289244 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-maas-api-tls\") pod \"3dbb737f-ccec-4959-af11-33486428f42f\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " Nov 03 19:33:53.289390 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.289287 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbb737f-ccec-4959-af11-33486428f42f-kube-api-access-5xltv\" (UniqueName: \"kubernetes.io/projected/3dbb737f-ccec-4959-af11-33486428f42f-kube-api-access-5xltv\") pod \"3dbb737f-ccec-4959-af11-33486428f42f\" (UID: \"3dbb737f-ccec-4959-af11-33486428f42f\") " Nov 03 19:33:53.290122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.290091 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbb737f-ccec-4959-af11-33486428f42f-openshift-service-ca" pod "3dbb737f-ccec-4959-af11-33486428f42f" (UID: "3dbb737f-ccec-4959-af11-33486428f42f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:53.292741 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.292701 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbb737f-ccec-4959-af11-33486428f42f-kube-api-access-5xltv" pod "3dbb737f-ccec-4959-af11-33486428f42f" (UID: "3dbb737f-ccec-4959-af11-33486428f42f"). InnerVolumeSpecName "kube-api-access-5xltv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:53.293116 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.293092 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-metrics-certs" pod "3dbb737f-ccec-4959-af11-33486428f42f" (UID: "3dbb737f-ccec-4959-af11-33486428f42f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:33:53.294606 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.294582 5019 generic.go:356] "Generic (PLEG): container finished" podID="3dbb737f-ccec-4959-af11-33486428f42f" containerID="2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e" exitCode=0 Nov 03 19:33:53.294756 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.294738 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" event={"ID":"3dbb737f-ccec-4959-af11-33486428f42f","Type":"ContainerDied","Data":"2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e"} Nov 03 19:33:53.295710 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.294794 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" event={"ID":"3dbb737f-ccec-4959-af11-33486428f42f","Type":"ContainerDied","Data":"a2df3e652c40aa7c5e5a01dc692bca720f9cb223108b625f4909c77920fe8add"} Nov 03 19:33:53.295710 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.294822 5019 scope.go:122] "RemoveContainer" containerID="2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e" Nov 03 19:33:53.295710 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.294984 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2" Nov 03 19:33:53.296986 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.296966 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-maas-api-tls" pod "3dbb737f-ccec-4959-af11-33486428f42f" (UID: "3dbb737f-ccec-4959-af11-33486428f42f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:33:53.327753 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.327728 5019 scope.go:122] "RemoveContainer" containerID="2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e" Nov 03 19:33:53.328040 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:33:53.328021 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e\": container with ID starting with 2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e not found: ID does not exist" containerID="2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e" Nov 03 19:33:53.328104 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.328047 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e"} err="failed to get container status \"2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e\": rpc error: code = NotFound desc = could not find container \"2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e\": container with ID starting with 2cecdc2cedf9ff28134591eb4dbefb5e4f37e82268d7452cbbe99594fafafc2e not found: ID does not exist" Nov 03 19:33:53.395782 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.392649 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbb737f-ccec-4959-af11-33486428f42f-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:53.395782 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.392691 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:53.395782 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.392715 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbb737f-ccec-4959-af11-33486428f42f-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:53.395782 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.392728 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xltv\" (UniqueName: \"kubernetes.io/projected/3dbb737f-ccec-4959-af11-33486428f42f-kube-api-access-5xltv\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:53.709103 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.709071 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2"] Nov 03 19:33:53.714445 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:53.714421 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-22ed3b-5c6fcdffd6-2j2z2"] Nov 03 19:33:54.150456 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:33:54.150380 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbb737f-ccec-4959-af11-33486428f42f" path="/var/lib/kubelet/pods/3dbb737f-ccec-4959-af11-33486428f42f/volumes" Nov 03 19:34:02.546730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:02.546693 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt"] Nov 03 19:34:02.547140 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:02.546947 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" containerID="cri-o://46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea" gracePeriod=30 Nov 03 19:34:04.759373 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:04.759338 5019 patch_prober.go:29] interesting pod/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:34:04.759859 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:04.759404 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:34:07.819549 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.819487 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:34:07.958341 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.958303 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk"] Nov 03 19:34:07.962937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.962907 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istiod-ca-cert\") pod \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " Nov 03 19:34:07.963086 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.962955 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-credential-socket\") pod \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " Nov 03 19:34:07.963086 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.963005 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-token\" (UniqueName: \"kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-token\") pod \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " Nov 03 19:34:07.963196 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.963096 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-certs\") pod \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " Nov 03 19:34:07.963359 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.963320 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-credential-socket" pod "0629a8f5-b876-46b6-bb0f-5d73ae38a489" (UID: "0629a8f5-b876-46b6-bb0f-5d73ae38a489"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:07.963443 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.963414 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-envoy\") pod \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " Nov 03 19:34:07.963495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.963434 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-certs" pod "0629a8f5-b876-46b6-bb0f-5d73ae38a489" (UID: "0629a8f5-b876-46b6-bb0f-5d73ae38a489"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:07.963495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.963461 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-data\") pod \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " Nov 03 19:34:07.963495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.963481 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istiod-ca-cert" pod "0629a8f5-b876-46b6-bb0f-5d73ae38a489" (UID: "0629a8f5-b876-46b6-bb0f-5d73ae38a489"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:34:07.963697 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.963522 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-kube-api-access-qtl9d\" (UniqueName: \"kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-kube-api-access-qtl9d\") pod \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " Nov 03 19:34:07.963697 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.963579 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-socket\") pod \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " Nov 03 19:34:07.963697 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.963621 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-podinfo\") pod \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\" (UID: \"0629a8f5-b876-46b6-bb0f-5d73ae38a489\") " Nov 03 19:34:07.964077 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.964056 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:07.964214 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.964082 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:07.964214 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.964116 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:07.964895 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.964858 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-data" pod "0629a8f5-b876-46b6-bb0f-5d73ae38a489" (UID: "0629a8f5-b876-46b6-bb0f-5d73ae38a489"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:07.965088 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.965056 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-socket" pod "0629a8f5-b876-46b6-bb0f-5d73ae38a489" (UID: "0629a8f5-b876-46b6-bb0f-5d73ae38a489"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:07.966242 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.966220 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-envoy" pod "0629a8f5-b876-46b6-bb0f-5d73ae38a489" (UID: "0629a8f5-b876-46b6-bb0f-5d73ae38a489"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:07.966375 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.966270 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-token" pod "0629a8f5-b876-46b6-bb0f-5d73ae38a489" (UID: "0629a8f5-b876-46b6-bb0f-5d73ae38a489"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:07.967201 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.967177 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-podinfo" pod "0629a8f5-b876-46b6-bb0f-5d73ae38a489" (UID: "0629a8f5-b876-46b6-bb0f-5d73ae38a489"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:34:07.967424 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.967397 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-kube-api-access-qtl9d" pod "0629a8f5-b876-46b6-bb0f-5d73ae38a489" (UID: "0629a8f5-b876-46b6-bb0f-5d73ae38a489"). InnerVolumeSpecName "kube-api-access-qtl9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:07.975134 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:07.975107 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl"] Nov 03 19:34:08.066118 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.066082 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:08.066118 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.066119 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:08.066382 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.066139 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtl9d\" (UniqueName: \"kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-kube-api-access-qtl9d\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:08.066382 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.066155 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0629a8f5-b876-46b6-bb0f-5d73ae38a489-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:08.066382 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.066170 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:08.066382 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.066184 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0629a8f5-b876-46b6-bb0f-5d73ae38a489-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:08.387853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.387816 5019 generic.go:356] "Generic (PLEG): container finished" podID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerID="46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea" exitCode=0 Nov 03 19:34:08.388044 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.388028 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" event={"ID":"0629a8f5-b876-46b6-bb0f-5d73ae38a489","Type":"ContainerDied","Data":"46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea"} Nov 03 19:34:08.388096 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.388083 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" event={"ID":"0629a8f5-b876-46b6-bb0f-5d73ae38a489","Type":"ContainerDied","Data":"1a549b59b0b94639515a48b4a1c099a08935a3c9b08bd61eb0b0d288d43239d5"} Nov 03 19:34:08.388135 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.388115 5019 scope.go:122] "RemoveContainer" containerID="46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea" Nov 03 19:34:08.388283 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.388265 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt" Nov 03 19:34:08.398242 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.398219 5019 scope.go:122] "RemoveContainer" containerID="7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93" Nov 03 19:34:08.409104 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.409087 5019 scope.go:122] "RemoveContainer" containerID="46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea" Nov 03 19:34:08.409457 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:34:08.409436 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea\": container with ID starting with 46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea not found: ID does not exist" containerID="46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea" Nov 03 19:34:08.409523 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.409466 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea"} err="failed to get container status \"46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea\": rpc error: code = NotFound desc = could not find container \"46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea\": container with ID starting with 46dfedfc5a0c7689842313a8d3b31f830d7992435dff337ac7ae048d994901ea not found: ID does not exist" Nov 03 19:34:08.409523 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.409486 5019 scope.go:122] "RemoveContainer" containerID="7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93" Nov 03 19:34:08.409763 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:34:08.409741 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93\": container with ID starting with 7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93 not found: ID does not exist" containerID="7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93" Nov 03 19:34:08.409812 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.409776 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93"} err="failed to get container status \"7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93\": rpc error: code = NotFound desc = could not find container \"7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93\": container with ID starting with 7b6289467234ffba4b3651c2f4b803800a940e1aea9cf86481d079175adc7f93 not found: ID does not exist" Nov 03 19:34:08.546529 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.546495 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt"] Nov 03 19:34:08.550220 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:08.550189 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-22ed3b-openshift-default-8547986d85-lhvwt"] Nov 03 19:34:10.158071 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:10.152021 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" path="/var/lib/kubelet/pods/0629a8f5-b876-46b6-bb0f-5d73ae38a489/volumes" Nov 03 19:34:11.296609 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:34:11.296571 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:34:13.054664 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:13.054630 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f"] Nov 03 19:34:13.055114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:13.054882 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerName="istio-proxy" containerID="cri-o://2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720" gracePeriod=30 Nov 03 19:34:18.320479 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.320403 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:34:18.379914 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.379886 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-token\" (UniqueName: \"kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-token\") pod \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " Nov 03 19:34:18.380092 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.379960 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-socket\") pod \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " Nov 03 19:34:18.380092 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.379998 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-credential-socket\") pod \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " Nov 03 19:34:18.380092 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.380024 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-data\") pod \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " Nov 03 19:34:18.380092 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.380055 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-podinfo\") pod \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " Nov 03 19:34:18.380340 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.380148 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-kube-api-access-h87z4\" (UniqueName: \"kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-kube-api-access-h87z4\") pod \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " Nov 03 19:34:18.380340 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.380177 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-envoy\") pod \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " Nov 03 19:34:18.380340 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.380242 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-certs\") pod \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " Nov 03 19:34:18.380340 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.380269 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istiod-ca-cert\") pod \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\" (UID: \"6c81f2ad-11c5-43a2-bde7-347f7e5f3715\") " Nov 03 19:34:18.380557 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.380338 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-socket" pod "6c81f2ad-11c5-43a2-bde7-347f7e5f3715" (UID: "6c81f2ad-11c5-43a2-bde7-347f7e5f3715"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:18.380557 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.380317 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-credential-socket" pod "6c81f2ad-11c5-43a2-bde7-347f7e5f3715" (UID: "6c81f2ad-11c5-43a2-bde7-347f7e5f3715"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:18.380557 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.380376 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-data" pod "6c81f2ad-11c5-43a2-bde7-347f7e5f3715" (UID: "6c81f2ad-11c5-43a2-bde7-347f7e5f3715"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:18.380734 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.380569 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-certs" pod "6c81f2ad-11c5-43a2-bde7-347f7e5f3715" (UID: "6c81f2ad-11c5-43a2-bde7-347f7e5f3715"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:18.389365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.381310 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istiod-ca-cert" pod "6c81f2ad-11c5-43a2-bde7-347f7e5f3715" (UID: "6c81f2ad-11c5-43a2-bde7-347f7e5f3715"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:34:18.389365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.382134 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:18.389365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.382161 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:18.389365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.382180 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:18.389365 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.382202 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:18.390944 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.390912 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-token" pod "6c81f2ad-11c5-43a2-bde7-347f7e5f3715" (UID: "6c81f2ad-11c5-43a2-bde7-347f7e5f3715"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:18.391095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.391076 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-envoy" pod "6c81f2ad-11c5-43a2-bde7-347f7e5f3715" (UID: "6c81f2ad-11c5-43a2-bde7-347f7e5f3715"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:18.395181 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.395155 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-kube-api-access-h87z4" pod "6c81f2ad-11c5-43a2-bde7-347f7e5f3715" (UID: "6c81f2ad-11c5-43a2-bde7-347f7e5f3715"). InnerVolumeSpecName "kube-api-access-h87z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:18.397020 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.396993 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-podinfo" pod "6c81f2ad-11c5-43a2-bde7-347f7e5f3715" (UID: "6c81f2ad-11c5-43a2-bde7-347f7e5f3715"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:34:18.447425 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.447394 5019 generic.go:356] "Generic (PLEG): container finished" podID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerID="2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720" exitCode=0 Nov 03 19:34:18.447554 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.447473 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" Nov 03 19:34:18.447554 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.447482 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" event={"ID":"6c81f2ad-11c5-43a2-bde7-347f7e5f3715","Type":"ContainerDied","Data":"2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720"} Nov 03 19:34:18.447626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.447566 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f" event={"ID":"6c81f2ad-11c5-43a2-bde7-347f7e5f3715","Type":"ContainerDied","Data":"e51f36a119ff7583cf3aeddeaafd92f70ce8eb5b0a13b8b5a803b52193562c59"} Nov 03 19:34:18.447626 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.447593 5019 scope.go:122] "RemoveContainer" containerID="2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720" Nov 03 19:34:18.457664 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.457642 5019 scope.go:122] "RemoveContainer" containerID="8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561" Nov 03 19:34:18.466854 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.466838 5019 scope.go:122] "RemoveContainer" containerID="2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720" Nov 03 19:34:18.467137 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:34:18.467116 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720\": container with ID starting with 2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720 not found: ID does not exist" containerID="2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720" Nov 03 19:34:18.467234 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.467148 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720"} err="failed to get container status \"2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720\": rpc error: code = NotFound desc = could not find container \"2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720\": container with ID starting with 2bff7035da0e8c17862ffe1e60006c956e7fc85121753ddce64e29d49e896720 not found: ID does not exist" Nov 03 19:34:18.467234 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.467170 5019 scope.go:122] "RemoveContainer" containerID="8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561" Nov 03 19:34:18.467452 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:34:18.467433 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561\": container with ID starting with 8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561 not found: ID does not exist" containerID="8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561" Nov 03 19:34:18.467525 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.467463 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561"} err="failed to get container status \"8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561\": rpc error: code = NotFound desc = could not find container \"8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561\": container with ID starting with 8e2c9cea2cfdb04a92bec4433481e1b43bf952dceffcc2135045da874859d561 not found: ID does not exist" Nov 03 19:34:18.483381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.483359 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:18.483488 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.483384 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:18.483488 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.483401 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h87z4\" (UniqueName: \"kubernetes.io/projected/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-kube-api-access-h87z4\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:18.483488 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.483416 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:18.483488 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.483430 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c81f2ad-11c5-43a2-bde7-347f7e5f3715-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:18.529356 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.529287 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc"] Nov 03 19:34:18.591102 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.591011 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f"] Nov 03 19:34:18.596902 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:18.596865 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-826a68-openshift-default-66cc858cd9-6tf6f"] Nov 03 19:34:20.149989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:20.149957 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" path="/var/lib/kubelet/pods/6c81f2ad-11c5-43a2-bde7-347f7e5f3715/volumes" Nov 03 19:34:22.985958 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:22.985914 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" containerID="cri-o://84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16" gracePeriod=45 Nov 03 19:34:22.996881 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:22.996836 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" containerID="cri-o://244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e" gracePeriod=45 Nov 03 19:34:23.273252 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.273222 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:34:23.307656 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.307633 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:34:23.343639 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.343615 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-home\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-home\") pod \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " Nov 03 19:34:23.343768 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.343701 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-dshm\") pod \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " Nov 03 19:34:23.343828 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.343766 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-model-cache\") pod \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " Nov 03 19:34:23.343828 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.343804 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tmp-dir\") pod \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " Nov 03 19:34:23.343941 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.343852 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-kube-api-access-p2kdp\" (UniqueName: \"kubernetes.io/projected/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-kube-api-access-p2kdp\") pod \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " Nov 03 19:34:23.343941 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.343883 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-home" pod "9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" (UID: "9b29c3f1-e002-4fcd-8ef8-3b1760d48ace"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:23.343941 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.343900 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tls-certs\") pod \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\" (UID: \"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace\") " Nov 03 19:34:23.344094 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.344056 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tmp-dir" pod "9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" (UID: "9b29c3f1-e002-4fcd-8ef8-3b1760d48ace"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:23.344094 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.344071 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-model-cache" pod "9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" (UID: "9b29c3f1-e002-4fcd-8ef8-3b1760d48ace"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:23.344408 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.344385 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-home\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:23.344495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.344414 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-model-cache\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:23.344495 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.344431 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tmp-dir\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:23.346028 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.346003 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-dshm" pod "9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" (UID: "9b29c3f1-e002-4fcd-8ef8-3b1760d48ace"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:23.346111 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.346065 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tls-certs" pod "9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" (UID: "9b29c3f1-e002-4fcd-8ef8-3b1760d48ace"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:23.346111 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.346091 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-kube-api-access-p2kdp" pod "9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" (UID: "9b29c3f1-e002-4fcd-8ef8-3b1760d48ace"). InnerVolumeSpecName "kube-api-access-p2kdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:23.445306 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.445271 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tls-certs\") pod \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " Nov 03 19:34:23.445534 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.445367 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b13be41-3746-4fc5-b7a5-2a2be4031df0-kube-api-access-j26n2\" (UniqueName: \"kubernetes.io/projected/6b13be41-3746-4fc5-b7a5-2a2be4031df0-kube-api-access-j26n2\") pod \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " Nov 03 19:34:23.445534 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.445411 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-model-cache\") pod \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " Nov 03 19:34:23.445534 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.445507 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-home\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-home\") pod \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " Nov 03 19:34:23.445717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.445557 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-dshm\") pod \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " Nov 03 19:34:23.445717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.445606 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tmp-dir\") pod \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\" (UID: \"6b13be41-3746-4fc5-b7a5-2a2be4031df0\") " Nov 03 19:34:23.445818 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.445764 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-home" pod "6b13be41-3746-4fc5-b7a5-2a2be4031df0" (UID: "6b13be41-3746-4fc5-b7a5-2a2be4031df0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:23.445941 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.445913 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tmp-dir" pod "6b13be41-3746-4fc5-b7a5-2a2be4031df0" (UID: "6b13be41-3746-4fc5-b7a5-2a2be4031df0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:23.446026 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.445959 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-model-cache" pod "6b13be41-3746-4fc5-b7a5-2a2be4031df0" (UID: "6b13be41-3746-4fc5-b7a5-2a2be4031df0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:23.446093 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.446080 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2kdp\" (UniqueName: \"kubernetes.io/projected/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-kube-api-access-p2kdp\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:23.446155 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.446102 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-tls-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:23.446155 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.446117 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-model-cache\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:23.446155 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.446131 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-home\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:23.446155 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.446144 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tmp-dir\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:23.446408 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.446157 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace-dshm\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:23.447597 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.447575 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tls-certs" pod "6b13be41-3746-4fc5-b7a5-2a2be4031df0" (UID: "6b13be41-3746-4fc5-b7a5-2a2be4031df0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:23.447806 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.447790 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-dshm" pod "6b13be41-3746-4fc5-b7a5-2a2be4031df0" (UID: "6b13be41-3746-4fc5-b7a5-2a2be4031df0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:23.447889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.447799 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b13be41-3746-4fc5-b7a5-2a2be4031df0-kube-api-access-j26n2" pod "6b13be41-3746-4fc5-b7a5-2a2be4031df0" (UID: "6b13be41-3746-4fc5-b7a5-2a2be4031df0"). InnerVolumeSpecName "kube-api-access-j26n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:23.477061 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.477028 5019 generic.go:356] "Generic (PLEG): container finished" podID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerID="244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e" exitCode=0 Nov 03 19:34:23.477226 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.477089 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" Nov 03 19:34:23.477226 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.477109 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" event={"ID":"6b13be41-3746-4fc5-b7a5-2a2be4031df0","Type":"ContainerDied","Data":"244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e"} Nov 03 19:34:23.477226 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.477161 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl" event={"ID":"6b13be41-3746-4fc5-b7a5-2a2be4031df0","Type":"ContainerDied","Data":"5db34208a06193bfba6c01ecd6257de4cda3e6f0c7e9f6d960709ffffc0bb1c6"} Nov 03 19:34:23.477226 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.477199 5019 scope.go:122] "RemoveContainer" containerID="244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e" Nov 03 19:34:23.479217 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.479195 5019 generic.go:356] "Generic (PLEG): container finished" podID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerID="84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16" exitCode=0 Nov 03 19:34:23.479363 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.479249 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" Nov 03 19:34:23.479496 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.479252 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" event={"ID":"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace","Type":"ContainerDied","Data":"84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16"} Nov 03 19:34:23.479567 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.479519 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk" event={"ID":"9b29c3f1-e002-4fcd-8ef8-3b1760d48ace","Type":"ContainerDied","Data":"171d174b239256fdb0ed3e6504b53c582d42181ae328f2ab2fbc35256bead0cb"} Nov 03 19:34:23.489738 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.489717 5019 scope.go:122] "RemoveContainer" containerID="244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e" Nov 03 19:34:23.490033 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:34:23.490010 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e\": container with ID starting with 244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e not found: ID does not exist" containerID="244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e" Nov 03 19:34:23.490097 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.490046 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e"} err="failed to get container status \"244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e\": rpc error: code = NotFound desc = could not find container \"244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e\": container with ID starting with 244b24ffdede771897d3a4c6b4b836431bb9f8eec5134e892e49d8c09425159e not found: ID does not exist" Nov 03 19:34:23.490097 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.490070 5019 scope.go:122] "RemoveContainer" containerID="84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16" Nov 03 19:34:23.502152 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.501839 5019 scope.go:122] "RemoveContainer" containerID="84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16" Nov 03 19:34:23.505773 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:34:23.502200 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16\": container with ID starting with 84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16 not found: ID does not exist" containerID="84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16" Nov 03 19:34:23.505773 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.502246 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16"} err="failed to get container status \"84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16\": rpc error: code = NotFound desc = could not find container \"84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16\": container with ID starting with 84e09c4b8da2d8056769fe2418a53077d3e1d3f0403d4e994097f9b2c0d9aa16 not found: ID does not exist" Nov 03 19:34:23.546713 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.546684 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b13be41-3746-4fc5-b7a5-2a2be4031df0-dshm\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:23.546713 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.546713 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b13be41-3746-4fc5-b7a5-2a2be4031df0-tls-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:23.546880 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.546725 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j26n2\" (UniqueName: \"kubernetes.io/projected/6b13be41-3746-4fc5-b7a5-2a2be4031df0-kube-api-access-j26n2\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:23.600406 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.600322 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk"] Nov 03 19:34:23.603894 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.603865 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/distinct-2-w6-eb6ef8-kserve-7f65897f48-vxdlk"] Nov 03 19:34:23.706532 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.706493 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl"] Nov 03 19:34:23.710584 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:23.710551 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/premium-w6-eb6ef8-kserve-84cf7d68f6-zrjvl"] Nov 03 19:34:24.149554 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:24.149523 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" path="/var/lib/kubelet/pods/6b13be41-3746-4fc5-b7a5-2a2be4031df0/volumes" Nov 03 19:34:24.149972 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:24.149917 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" path="/var/lib/kubelet/pods/9b29c3f1-e002-4fcd-8ef8-3b1760d48ace/volumes" Nov 03 19:34:29.855752 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:29.855723 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:34:29.861086 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:29.861057 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:34:33.551164 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.551092 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" containerID="cri-o://11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8" gracePeriod=45 Nov 03 19:34:33.829506 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.829483 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:34:33.955401 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.955368 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90d072fe-3d22-4a06-9227-87069cf3e96c-tls-certs\" (UniqueName: \"kubernetes.io/secret/90d072fe-3d22-4a06-9227-87069cf3e96c-tls-certs\") pod \"90d072fe-3d22-4a06-9227-87069cf3e96c\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " Nov 03 19:34:33.955610 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.955586 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-model-cache\") pod \"90d072fe-3d22-4a06-9227-87069cf3e96c\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " Nov 03 19:34:33.955712 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.955679 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-home\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-home\") pod \"90d072fe-3d22-4a06-9227-87069cf3e96c\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " Nov 03 19:34:33.955781 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.955708 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-dshm\") pod \"90d072fe-3d22-4a06-9227-87069cf3e96c\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " Nov 03 19:34:33.955919 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.955889 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-model-cache" pod "90d072fe-3d22-4a06-9227-87069cf3e96c" (UID: "90d072fe-3d22-4a06-9227-87069cf3e96c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.956066 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.955943 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-home" pod "90d072fe-3d22-4a06-9227-87069cf3e96c" (UID: "90d072fe-3d22-4a06-9227-87069cf3e96c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.956143 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.956079 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d072fe-3d22-4a06-9227-87069cf3e96c-kube-api-access-5bk7z\" (UniqueName: \"kubernetes.io/projected/90d072fe-3d22-4a06-9227-87069cf3e96c-kube-api-access-5bk7z\") pod \"90d072fe-3d22-4a06-9227-87069cf3e96c\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " Nov 03 19:34:33.956143 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.956115 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-tmp-dir\") pod \"90d072fe-3d22-4a06-9227-87069cf3e96c\" (UID: \"90d072fe-3d22-4a06-9227-87069cf3e96c\") " Nov 03 19:34:33.956610 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.956579 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-tmp-dir" pod "90d072fe-3d22-4a06-9227-87069cf3e96c" (UID: "90d072fe-3d22-4a06-9227-87069cf3e96c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.956716 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.956605 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-model-cache\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:33.956716 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.956626 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-home\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:33.958275 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.958254 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-dshm" pod "90d072fe-3d22-4a06-9227-87069cf3e96c" (UID: "90d072fe-3d22-4a06-9227-87069cf3e96c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:34:33.960974 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.959423 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d072fe-3d22-4a06-9227-87069cf3e96c-kube-api-access-5bk7z" pod "90d072fe-3d22-4a06-9227-87069cf3e96c" (UID: "90d072fe-3d22-4a06-9227-87069cf3e96c"). InnerVolumeSpecName "kube-api-access-5bk7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:33.961444 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:33.961411 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d072fe-3d22-4a06-9227-87069cf3e96c-tls-certs" pod "90d072fe-3d22-4a06-9227-87069cf3e96c" (UID: "90d072fe-3d22-4a06-9227-87069cf3e96c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:34.058161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.058079 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d072fe-3d22-4a06-9227-87069cf3e96c-tls-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.058161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.058109 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-dshm\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.058161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.058126 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bk7z\" (UniqueName: \"kubernetes.io/projected/90d072fe-3d22-4a06-9227-87069cf3e96c-kube-api-access-5bk7z\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.058161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.058139 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d072fe-3d22-4a06-9227-87069cf3e96c-tmp-dir\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.542970 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.542940 5019 generic.go:356] "Generic (PLEG): container finished" podID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerID="11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8" exitCode=0 Nov 03 19:34:34.543144 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.542975 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" event={"ID":"90d072fe-3d22-4a06-9227-87069cf3e96c","Type":"ContainerDied","Data":"11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8"} Nov 03 19:34:34.543144 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.542988 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" Nov 03 19:34:34.543144 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.543005 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc" event={"ID":"90d072fe-3d22-4a06-9227-87069cf3e96c","Type":"ContainerDied","Data":"71274d0b25989b6f12596076b2c13a2e3ab55b87708e82904e68b4f029e088ea"} Nov 03 19:34:34.543144 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.543022 5019 scope.go:122] "RemoveContainer" containerID="11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8" Nov 03 19:34:34.552195 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.552016 5019 scope.go:122] "RemoveContainer" containerID="11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8" Nov 03 19:34:34.552483 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:34:34.552302 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8\": container with ID starting with 11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8 not found: ID does not exist" containerID="11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8" Nov 03 19:34:34.552483 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.552351 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8"} err="failed to get container status \"11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8\": rpc error: code = NotFound desc = could not find container \"11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8\": container with ID starting with 11e7bc488b047bf34456c8706812ce1853586295c166a363e5e000028b76ece8 not found: ID does not exist" Nov 03 19:34:34.696775 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.696742 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc"] Nov 03 19:34:34.701457 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:34.701427 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fa2878/distinct-w3-fa2878-kserve-74bc5dc6cd-59wgc"] Nov 03 19:34:36.149586 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:36.149506 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" path="/var/lib/kubelet/pods/90d072fe-3d22-4a06-9227-87069cf3e96c/volumes" Nov 03 19:34:41.296833 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:34:41.296789 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:34:49.573071 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:49.573039 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j"] Nov 03 19:34:49.573493 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:49.573285 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" podUID="5f7d428b-6291-4714-8848-5ada8ce9f341" containerName="maas-api" containerID="cri-o://af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2" gracePeriod=30 Nov 03 19:34:49.865771 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:49.865746 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 19:34:49.949965 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:49.949912 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7d428b-6291-4714-8848-5ada8ce9f341-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7d428b-6291-4714-8848-5ada8ce9f341-openshift-service-ca\") pod \"5f7d428b-6291-4714-8848-5ada8ce9f341\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " Nov 03 19:34:49.950163 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:49.949972 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7d428b-6291-4714-8848-5ada8ce9f341-kube-api-access-f2jv4\" (UniqueName: \"kubernetes.io/projected/5f7d428b-6291-4714-8848-5ada8ce9f341-kube-api-access-f2jv4\") pod \"5f7d428b-6291-4714-8848-5ada8ce9f341\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " Nov 03 19:34:49.950163 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:49.950061 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-maas-api-tls\") pod \"5f7d428b-6291-4714-8848-5ada8ce9f341\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " Nov 03 19:34:49.950163 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:49.950118 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-metrics-certs\") pod \"5f7d428b-6291-4714-8848-5ada8ce9f341\" (UID: \"5f7d428b-6291-4714-8848-5ada8ce9f341\") " Nov 03 19:34:49.950628 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:49.950392 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7d428b-6291-4714-8848-5ada8ce9f341-openshift-service-ca" pod "5f7d428b-6291-4714-8848-5ada8ce9f341" (UID: "5f7d428b-6291-4714-8848-5ada8ce9f341"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:34:49.950628 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:49.950590 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7d428b-6291-4714-8848-5ada8ce9f341-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:49.954412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:49.954381 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-metrics-certs" pod "5f7d428b-6291-4714-8848-5ada8ce9f341" (UID: "5f7d428b-6291-4714-8848-5ada8ce9f341"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:49.955164 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:49.955135 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7d428b-6291-4714-8848-5ada8ce9f341-kube-api-access-f2jv4" pod "5f7d428b-6291-4714-8848-5ada8ce9f341" (UID: "5f7d428b-6291-4714-8848-5ada8ce9f341"). InnerVolumeSpecName "kube-api-access-f2jv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:49.956038 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:49.955400 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-maas-api-tls" pod "5f7d428b-6291-4714-8848-5ada8ce9f341" (UID: "5f7d428b-6291-4714-8848-5ada8ce9f341"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:50.051452 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:50.051417 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:50.051452 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:50.051448 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2jv4\" (UniqueName: \"kubernetes.io/projected/5f7d428b-6291-4714-8848-5ada8ce9f341-kube-api-access-f2jv4\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:50.051452 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:50.051458 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f7d428b-6291-4714-8848-5ada8ce9f341-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:50.641089 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:50.641056 5019 generic.go:356] "Generic (PLEG): container finished" podID="5f7d428b-6291-4714-8848-5ada8ce9f341" containerID="af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2" exitCode=0 Nov 03 19:34:50.641595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:50.641099 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" Nov 03 19:34:50.641595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:50.641116 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" event={"ID":"5f7d428b-6291-4714-8848-5ada8ce9f341","Type":"ContainerDied","Data":"af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2"} Nov 03 19:34:50.641595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:50.641163 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j" event={"ID":"5f7d428b-6291-4714-8848-5ada8ce9f341","Type":"ContainerDied","Data":"7a67a10d0cc91aad928e5255c349a7d591b2f9b7c7e52e6c5dd56a8529ef4d66"} Nov 03 19:34:50.641595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:50.641186 5019 scope.go:122] "RemoveContainer" containerID="af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2" Nov 03 19:34:50.650214 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:50.650193 5019 scope.go:122] "RemoveContainer" containerID="af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2" Nov 03 19:34:50.650501 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:34:50.650483 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2\": container with ID starting with af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2 not found: ID does not exist" containerID="af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2" Nov 03 19:34:50.650550 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:50.650509 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2"} err="failed to get container status \"af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2\": rpc error: code = NotFound desc = could not find container \"af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2\": container with ID starting with af4521bbbba451038e4a561594aebb342cf0df9a5e1ec42bc120cdcafa265dc2 not found: ID does not exist" Nov 03 19:34:50.860474 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:50.860440 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j"] Nov 03 19:34:50.906290 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:50.906217 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fa2878-68bf5c944c-9g55j"] Nov 03 19:34:52.151152 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.151122 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7d428b-6291-4714-8848-5ada8ce9f341" path="/var/lib/kubelet/pods/5f7d428b-6291-4714-8848-5ada8ce9f341/volumes" Nov 03 19:34:52.695100 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695069 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4"] Nov 03 19:34:52.695584 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695570 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" Nov 03 19:34:52.695637 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695586 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddca4421-6d98-415e-9848-80ec181cb4ae" containerName="istio-proxy" Nov 03 19:34:52.695637 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695599 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerName="istio-proxy" Nov 03 19:34:52.695637 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695604 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerName="istio-proxy" Nov 03 19:34:52.695637 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695615 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" Nov 03 19:34:52.695637 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695620 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" Nov 03 19:34:52.695637 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695629 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerName="istio-proxy" Nov 03 19:34:52.695637 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695633 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerName="istio-proxy" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695641 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b" containerName="maas-api" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695646 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b" containerName="maas-api" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695661 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695665 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695673 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4d5e2740-ab79-4f6e-97cb-bac3339fde21" containerName="maas-api" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695678 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5e2740-ab79-4f6e-97cb-bac3339fde21" containerName="maas-api" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695685 5019 cpu_manager.go:414] "containerMap: removing container" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695689 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695695 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3dbb737f-ccec-4959-af11-33486428f42f" containerName="maas-api" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695699 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbb737f-ccec-4959-af11-33486428f42f" containerName="maas-api" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695705 5019 cpu_manager.go:414] "containerMap: removing container" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695710 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695716 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695723 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695733 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5f7d428b-6291-4714-8848-5ada8ce9f341" containerName="maas-api" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695737 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7d428b-6291-4714-8848-5ada8ce9f341" containerName="maas-api" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695747 5019 cpu_manager.go:414] "containerMap: removing container" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695751 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695760 5019 cpu_manager.go:414] "containerMap: removing container" podUID="641da692-25ae-4e05-9e06-2417ab73918d" containerName="istio-proxy" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695764 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641da692-25ae-4e05-9e06-2417ab73918d" containerName="istio-proxy" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695770 5019 cpu_manager.go:414] "containerMap: removing container" podUID="81073a4c-136f-485e-91ec-329f36057350" containerName="maas-api" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695773 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81073a4c-136f-485e-91ec-329f36057350" containerName="maas-api" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695784 5019 cpu_manager.go:414] "containerMap: removing container" podUID="641da692-25ae-4e05-9e06-2417ab73918d" containerName="istio-proxy" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695788 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641da692-25ae-4e05-9e06-2417ab73918d" containerName="istio-proxy" Nov 03 19:34:52.695853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695855 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7d428b-6291-4714-8848-5ada8ce9f341" containerName="maas-api" Nov 03 19:34:52.696623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695865 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d072fe-3d22-4a06-9227-87069cf3e96c" containerName="main" Nov 03 19:34:52.696623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695871 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" Nov 03 19:34:52.696623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695877 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="641da692-25ae-4e05-9e06-2417ab73918d" containerName="istio-proxy" Nov 03 19:34:52.696623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695881 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="81073a4c-136f-485e-91ec-329f36057350" containerName="maas-api" Nov 03 19:34:52.696623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695887 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerName="istio-proxy" Nov 03 19:34:52.696623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695891 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b29c3f1-e002-4fcd-8ef8-3b1760d48ace" containerName="main" Nov 03 19:34:52.696623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695898 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99aaf9c-4f4e-4e1d-a1aa-a6b9c4f1b12b" containerName="maas-api" Nov 03 19:34:52.696623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695905 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b13be41-3746-4fc5-b7a5-2a2be4031df0" containerName="main" Nov 03 19:34:52.696623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695910 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5e2740-ab79-4f6e-97cb-bac3339fde21" containerName="maas-api" Nov 03 19:34:52.696623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695914 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" Nov 03 19:34:52.696623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695919 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbb737f-ccec-4959-af11-33486428f42f" containerName="maas-api" Nov 03 19:34:52.696623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.695924 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="501946c2-6e3d-4fb2-9123-e105014fb260" containerName="istio-proxy" Nov 03 19:34:52.700669 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.700649 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.749576 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.749536 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fa2878-openshift-default-dockercfg-hfvcf\"" Nov 03 19:34:52.781616 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.781578 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.781805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.781626 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.781805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.781731 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.781950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.781824 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386109e8-f361-4d97-b76a-f3d84e85a533-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.781950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.781866 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.781950 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.781939 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.782116 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.782065 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386109e8-f361-4d97-b76a-f3d84e85a533-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.782116 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.782106 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8f9q\" (UniqueName: \"kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-kube-api-access-q8f9q\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.782234 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.782134 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.799933 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.799888 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4"] Nov 03 19:34:52.883239 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.883201 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.883463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.883275 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386109e8-f361-4d97-b76a-f3d84e85a533-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.883463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.883299 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8f9q\" (UniqueName: \"kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-kube-api-access-q8f9q\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.883463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.883317 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.883463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.883426 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.883463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.883459 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.883751 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.883505 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.883868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.883647 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386109e8-f361-4d97-b76a-f3d84e85a533-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.883962 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.883908 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.883962 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.883848 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.884077 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.883908 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.884077 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.884039 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.884209 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.884189 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.884521 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.884501 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386109e8-f361-4d97-b76a-f3d84e85a533-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.885698 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.885677 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.886179 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.886165 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386109e8-f361-4d97-b76a-f3d84e85a533-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.941193 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.941156 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:52.941402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:52.941358 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8f9q\" (UniqueName: \"kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-kube-api-access-q8f9q\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:53.014167 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:53.014083 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:53.435439 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:53.435410 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4"] Nov 03 19:34:53.437265 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:34:53.437235 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386109e8_f361_4d97_b76a_f3d84e85a533.slice/crio-d3a50452a14ae0ce8cafc83b3b9c98f12e5ac84d37361eadb10185657ffc04a7 WatchSource:0}: Error finding container d3a50452a14ae0ce8cafc83b3b9c98f12e5ac84d37361eadb10185657ffc04a7: Status 404 returned error can't find the container with id d3a50452a14ae0ce8cafc83b3b9c98f12e5ac84d37361eadb10185657ffc04a7 Nov 03 19:34:53.439591 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:53.439561 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:34:53.439674 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:53.439624 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:34:53.439674 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:53.439658 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:34:53.659502 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:53.659469 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" event={"ID":"386109e8-f361-4d97-b76a-f3d84e85a533","Type":"ContainerStarted","Data":"dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed"} Nov 03 19:34:53.659675 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:53.659523 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" event={"ID":"386109e8-f361-4d97-b76a-f3d84e85a533","Type":"ContainerStarted","Data":"d3a50452a14ae0ce8cafc83b3b9c98f12e5ac84d37361eadb10185657ffc04a7"} Nov 03 19:34:53.775564 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:53.775448 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" podStartSLOduration=1.775427356 podStartE2EDuration="1.775427356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:53.772028416 +0000 UTC m=+3027.744589603" watchObservedRunningTime="2026-11-03 19:34:53.775427356 +0000 UTC m=+3027.747988540" Nov 03 19:34:54.015278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:54.015240 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:34:54.017090 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:54.017063 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:34:54.017221 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:54.017111 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" podUID="386109e8-f361-4d97-b76a-f3d84e85a533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:34:55.014788 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:55.014745 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:34:55.015319 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:55.014798 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" podUID="386109e8-f361-4d97-b76a-f3d84e85a533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:34:56.014990 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:56.014950 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:34:56.015423 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:56.015010 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" podUID="386109e8-f361-4d97-b76a-f3d84e85a533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:34:57.014697 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:57.014661 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:34:57.014880 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:57.014725 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" podUID="386109e8-f361-4d97-b76a-f3d84e85a533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:34:58.014734 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:58.014703 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:34:58.015129 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:58.014762 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" podUID="386109e8-f361-4d97-b76a-f3d84e85a533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:34:59.015113 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:59.015072 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:34:59.015658 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:59.015140 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" podUID="386109e8-f361-4d97-b76a-f3d84e85a533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:34:59.027029 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:59.027000 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4"] Nov 03 19:34:59.027402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:34:59.027373 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" podUID="386109e8-f361-4d97-b76a-f3d84e85a533" containerName="istio-proxy" containerID="cri-o://dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed" gracePeriod=30 Nov 03 19:35:06.389597 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.389568 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:35:06.530015 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.529924 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-istio-token\" (UniqueName: \"kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-istio-token\") pod \"386109e8-f361-4d97-b76a-f3d84e85a533\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " Nov 03 19:35:06.530187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530020 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386109e8-f361-4d97-b76a-f3d84e85a533-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386109e8-f361-4d97-b76a-f3d84e85a533-istiod-ca-cert\") pod \"386109e8-f361-4d97-b76a-f3d84e85a533\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " Nov 03 19:35:06.530187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530074 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-data\") pod \"386109e8-f361-4d97-b76a-f3d84e85a533\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " Nov 03 19:35:06.530187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530114 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-certs\") pod \"386109e8-f361-4d97-b76a-f3d84e85a533\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " Nov 03 19:35:06.530187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530143 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-credential-socket\") pod \"386109e8-f361-4d97-b76a-f3d84e85a533\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " Nov 03 19:35:06.530434 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530218 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-kube-api-access-q8f9q\" (UniqueName: \"kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-kube-api-access-q8f9q\") pod \"386109e8-f361-4d97-b76a-f3d84e85a533\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " Nov 03 19:35:06.530434 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530259 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/386109e8-f361-4d97-b76a-f3d84e85a533-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386109e8-f361-4d97-b76a-f3d84e85a533-istio-podinfo\") pod \"386109e8-f361-4d97-b76a-f3d84e85a533\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " Nov 03 19:35:06.530555 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530483 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-socket\") pod \"386109e8-f361-4d97-b76a-f3d84e85a533\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " Nov 03 19:35:06.530623 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530479 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-data" pod "386109e8-f361-4d97-b76a-f3d84e85a533" (UID: "386109e8-f361-4d97-b76a-f3d84e85a533"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:35:06.530682 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530462 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-certs" pod "386109e8-f361-4d97-b76a-f3d84e85a533" (UID: "386109e8-f361-4d97-b76a-f3d84e85a533"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:35:06.530682 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530627 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-envoy\") pod \"386109e8-f361-4d97-b76a-f3d84e85a533\" (UID: \"386109e8-f361-4d97-b76a-f3d84e85a533\") " Nov 03 19:35:06.530792 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530508 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-credential-socket" pod "386109e8-f361-4d97-b76a-f3d84e85a533" (UID: "386109e8-f361-4d97-b76a-f3d84e85a533"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:35:06.530792 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530568 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386109e8-f361-4d97-b76a-f3d84e85a533-istiod-ca-cert" pod "386109e8-f361-4d97-b76a-f3d84e85a533" (UID: "386109e8-f361-4d97-b76a-f3d84e85a533"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:35:06.530792 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.530678 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-socket" pod "386109e8-f361-4d97-b76a-f3d84e85a533" (UID: "386109e8-f361-4d97-b76a-f3d84e85a533"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:35:06.531064 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.531040 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386109e8-f361-4d97-b76a-f3d84e85a533-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:06.531132 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.531069 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:06.531132 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.531084 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:06.531132 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.531100 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:06.531132 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.531115 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:06.532409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.532381 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-kube-api-access-q8f9q" pod "386109e8-f361-4d97-b76a-f3d84e85a533" (UID: "386109e8-f361-4d97-b76a-f3d84e85a533"). InnerVolumeSpecName "kube-api-access-q8f9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:35:06.532409 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.532397 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-istio-token" pod "386109e8-f361-4d97-b76a-f3d84e85a533" (UID: "386109e8-f361-4d97-b76a-f3d84e85a533"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:35:06.532558 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.532544 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-envoy" pod "386109e8-f361-4d97-b76a-f3d84e85a533" (UID: "386109e8-f361-4d97-b76a-f3d84e85a533"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:35:06.532666 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.532651 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/386109e8-f361-4d97-b76a-f3d84e85a533-istio-podinfo" pod "386109e8-f361-4d97-b76a-f3d84e85a533" (UID: "386109e8-f361-4d97-b76a-f3d84e85a533"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:35:06.632600 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.632568 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8f9q\" (UniqueName: \"kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-kube-api-access-q8f9q\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:06.632600 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.632595 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386109e8-f361-4d97-b76a-f3d84e85a533-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:06.632600 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.632604 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386109e8-f361-4d97-b76a-f3d84e85a533-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:06.632851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.632615 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386109e8-f361-4d97-b76a-f3d84e85a533-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:06.758979 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.758938 5019 generic.go:356] "Generic (PLEG): container finished" podID="386109e8-f361-4d97-b76a-f3d84e85a533" containerID="dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed" exitCode=0 Nov 03 19:35:06.759158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.758991 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" Nov 03 19:35:06.759158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.759024 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" event={"ID":"386109e8-f361-4d97-b76a-f3d84e85a533","Type":"ContainerDied","Data":"dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed"} Nov 03 19:35:06.759158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.759083 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4" event={"ID":"386109e8-f361-4d97-b76a-f3d84e85a533","Type":"ContainerDied","Data":"d3a50452a14ae0ce8cafc83b3b9c98f12e5ac84d37361eadb10185657ffc04a7"} Nov 03 19:35:06.759158 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.759109 5019 scope.go:122] "RemoveContainer" containerID="dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed" Nov 03 19:35:06.769067 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.769044 5019 scope.go:122] "RemoveContainer" containerID="dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed" Nov 03 19:35:06.769377 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:35:06.769352 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed\": container with ID starting with dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed not found: ID does not exist" containerID="dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed" Nov 03 19:35:06.769463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.769390 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed"} err="failed to get container status \"dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed\": rpc error: code = NotFound desc = could not find container \"dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed\": container with ID starting with dce97202e3feb831c67fc40374514b68cef9c56f2874e9c5029c90a60bb936ed not found: ID does not exist" Nov 03 19:35:06.914298 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.914216 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4"] Nov 03 19:35:06.919173 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:06.919137 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jrlj4"] Nov 03 19:35:08.151759 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:35:08.151726 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386109e8-f361-4d97-b76a-f3d84e85a533" path="/var/lib/kubelet/pods/386109e8-f361-4d97-b76a-f3d84e85a533/volumes" Nov 03 19:35:11.298762 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:35:11.298723 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:35:41.297537 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:35:41.297498 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:36:11.297570 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:36:11.297480 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:36:41.317062 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:36:41.317019 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:36:43.265241 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:43.265207 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f"] Nov 03 19:36:43.265697 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:43.265484 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" podUID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" containerName="istio-proxy" containerID="cri-o://249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded" gracePeriod=30 Nov 03 19:36:43.313500 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:43.313459 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26"] Nov 03 19:36:43.313830 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:43.313778 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" containerName="istio-proxy" containerID="cri-o://eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1" gracePeriod=30 Nov 03 19:36:47.873622 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:47.873589 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6zg26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:36:47.874046 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:47.873654 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:36:48.575971 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.575942 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:36:48.612131 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.612103 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:36:48.615196 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615174 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-podinfo\") pod \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " Nov 03 19:36:48.615282 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615237 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-credential-socket\") pod \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " Nov 03 19:36:48.615282 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615262 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-socket\") pod \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " Nov 03 19:36:48.615417 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615287 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-kube-api-access-lfcjr\" (UniqueName: \"kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-kube-api-access-lfcjr\") pod \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " Nov 03 19:36:48.615417 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615359 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-envoy\") pod \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " Nov 03 19:36:48.615417 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615402 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-token\" (UniqueName: \"kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-token\") pod \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " Nov 03 19:36:48.615568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615460 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-data\") pod \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " Nov 03 19:36:48.615568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615496 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-credential-socket" pod "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" (UID: "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:48.615568 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615540 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istiod-ca-cert\") pod \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " Nov 03 19:36:48.615725 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615567 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-certs\") pod \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\" (UID: \"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8\") " Nov 03 19:36:48.615725 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615594 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-socket" pod "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" (UID: "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:48.615868 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615828 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-data" pod "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" (UID: "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:48.615932 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.615913 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istiod-ca-cert" pod "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" (UID: "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:36:48.616050 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.616028 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.616167 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.616052 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.616167 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.616067 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.616167 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.616080 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.616167 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.616072 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-certs" pod "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" (UID: "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:48.617805 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.617780 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-kube-api-access-lfcjr" pod "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" (UID: "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8"). InnerVolumeSpecName "kube-api-access-lfcjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:36:48.617976 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.617956 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-podinfo" pod "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" (UID: "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:36:48.618102 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.618083 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-envoy" pod "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" (UID: "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:48.618360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.618339 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-token" pod "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" (UID: "cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:36:48.717295 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717259 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-data\") pod \"ad3ba842-d764-4169-90fc-364512d8ebca\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " Nov 03 19:36:48.717531 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717317 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-kube-api-access-kzljp\" (UniqueName: \"kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-kube-api-access-kzljp\") pod \"ad3ba842-d764-4169-90fc-364512d8ebca\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " Nov 03 19:36:48.717531 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717383 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-credential-socket\") pod \"ad3ba842-d764-4169-90fc-364512d8ebca\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " Nov 03 19:36:48.717531 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717429 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-envoy\") pod \"ad3ba842-d764-4169-90fc-364512d8ebca\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " Nov 03 19:36:48.717531 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717463 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3ba842-d764-4169-90fc-364512d8ebca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3ba842-d764-4169-90fc-364512d8ebca-istiod-ca-cert\") pod \"ad3ba842-d764-4169-90fc-364512d8ebca\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " Nov 03 19:36:48.717531 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717492 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-socket\") pod \"ad3ba842-d764-4169-90fc-364512d8ebca\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " Nov 03 19:36:48.717818 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717592 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-istio-token\" (UniqueName: \"kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-istio-token\") pod \"ad3ba842-d764-4169-90fc-364512d8ebca\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " Nov 03 19:36:48.717818 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717619 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-data" pod "ad3ba842-d764-4169-90fc-364512d8ebca" (UID: "ad3ba842-d764-4169-90fc-364512d8ebca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:48.717818 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717760 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-certs\") pod \"ad3ba842-d764-4169-90fc-364512d8ebca\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " Nov 03 19:36:48.717818 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717773 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-credential-socket" pod "ad3ba842-d764-4169-90fc-364512d8ebca" (UID: "ad3ba842-d764-4169-90fc-364512d8ebca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:48.718044 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717811 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad3ba842-d764-4169-90fc-364512d8ebca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3ba842-d764-4169-90fc-364512d8ebca-istio-podinfo\") pod \"ad3ba842-d764-4169-90fc-364512d8ebca\" (UID: \"ad3ba842-d764-4169-90fc-364512d8ebca\") " Nov 03 19:36:48.718044 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717849 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-socket" pod "ad3ba842-d764-4169-90fc-364512d8ebca" (UID: "ad3ba842-d764-4169-90fc-364512d8ebca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:48.718044 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.717870 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3ba842-d764-4169-90fc-364512d8ebca-istiod-ca-cert" pod "ad3ba842-d764-4169-90fc-364512d8ebca" (UID: "ad3ba842-d764-4169-90fc-364512d8ebca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:36:48.718181 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.718114 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-certs" pod "ad3ba842-d764-4169-90fc-364512d8ebca" (UID: "ad3ba842-d764-4169-90fc-364512d8ebca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:48.718276 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.718256 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.718364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.718283 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfcjr\" (UniqueName: \"kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-kube-api-access-lfcjr\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.718364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.718299 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.718364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.718313 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.718364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.718342 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-data\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.718364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.718356 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-credential-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.718557 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.718370 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3ba842-d764-4169-90fc-364512d8ebca-istiod-ca-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.718557 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.718384 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-workload-socket\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.718557 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.718399 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-workload-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.718557 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.718413 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.719784 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.719762 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-envoy" pod "ad3ba842-d764-4169-90fc-364512d8ebca" (UID: "ad3ba842-d764-4169-90fc-364512d8ebca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:48.719864 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.719795 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-istio-token" pod "ad3ba842-d764-4169-90fc-364512d8ebca" (UID: "ad3ba842-d764-4169-90fc-364512d8ebca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:36:48.720125 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.720107 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-kube-api-access-kzljp" pod "ad3ba842-d764-4169-90fc-364512d8ebca" (UID: "ad3ba842-d764-4169-90fc-364512d8ebca"). InnerVolumeSpecName "kube-api-access-kzljp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:36:48.720125 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.720113 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad3ba842-d764-4169-90fc-364512d8ebca-istio-podinfo" pod "ad3ba842-d764-4169-90fc-364512d8ebca" (UID: "ad3ba842-d764-4169-90fc-364512d8ebca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:36:48.819360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.819241 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3ba842-d764-4169-90fc-364512d8ebca-istio-podinfo\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.819360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.819274 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzljp\" (UniqueName: \"kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-kube-api-access-kzljp\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.819360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.819286 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3ba842-d764-4169-90fc-364512d8ebca-istio-envoy\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.819360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:48.819300 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3ba842-d764-4169-90fc-364512d8ebca-istio-token\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:49.330488 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.330451 5019 generic.go:356] "Generic (PLEG): container finished" podID="ad3ba842-d764-4169-90fc-364512d8ebca" containerID="eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1" exitCode=0 Nov 03 19:36:49.330978 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.330497 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" Nov 03 19:36:49.330978 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.330538 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" event={"ID":"ad3ba842-d764-4169-90fc-364512d8ebca","Type":"ContainerDied","Data":"eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1"} Nov 03 19:36:49.330978 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.330596 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26" event={"ID":"ad3ba842-d764-4169-90fc-364512d8ebca","Type":"ContainerDied","Data":"774ae79b46d4f2958d26673b9352da0cb1278ed3670585ecd94f753a351784b5"} Nov 03 19:36:49.330978 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.330626 5019 scope.go:122] "RemoveContainer" containerID="eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1" Nov 03 19:36:49.331988 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.331966 5019 generic.go:356] "Generic (PLEG): container finished" podID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" containerID="249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded" exitCode=0 Nov 03 19:36:49.332082 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.332004 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" Nov 03 19:36:49.332082 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.332038 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" event={"ID":"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8","Type":"ContainerDied","Data":"249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded"} Nov 03 19:36:49.332082 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.332080 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f" event={"ID":"cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8","Type":"ContainerDied","Data":"aecd6f67e4436c2f5ac13b3ba8adcf3f832c7b29413233e599038f456bfb0fc9"} Nov 03 19:36:49.340695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.340678 5019 scope.go:122] "RemoveContainer" containerID="eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1" Nov 03 19:36:49.340982 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:36:49.340961 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1\": container with ID starting with eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1 not found: ID does not exist" containerID="eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1" Nov 03 19:36:49.341036 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.340991 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1"} err="failed to get container status \"eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1\": rpc error: code = NotFound desc = could not find container \"eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1\": container with ID starting with eade675abbdc956aa791f8d2482eed71dde45d8029df00311d5a91e695c2cdb1 not found: ID does not exist" Nov 03 19:36:49.341036 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.341015 5019 scope.go:122] "RemoveContainer" containerID="249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded" Nov 03 19:36:49.349562 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.349547 5019 scope.go:122] "RemoveContainer" containerID="249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded" Nov 03 19:36:49.349850 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:36:49.349832 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded\": container with ID starting with 249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded not found: ID does not exist" containerID="249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded" Nov 03 19:36:49.349913 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.349859 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded"} err="failed to get container status \"249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded\": rpc error: code = NotFound desc = could not find container \"249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded\": container with ID starting with 249e041f38517cc0267b9fa066b52837caa355ad74f60c56aa1d38df209caded not found: ID does not exist" Nov 03 19:36:49.486617 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.486580 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f"] Nov 03 19:36:49.493785 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.493754 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gmc4f"] Nov 03 19:36:49.595783 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.595703 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26"] Nov 03 19:36:49.599660 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:49.599623 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6zg26"] Nov 03 19:36:50.151556 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.151527 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" path="/var/lib/kubelet/pods/ad3ba842-d764-4169-90fc-364512d8ebca/volumes" Nov 03 19:36:50.151989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.151975 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" path="/var/lib/kubelet/pods/cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8/volumes" Nov 03 19:36:50.600187 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.600158 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f"] Nov 03 19:36:50.600687 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.600369 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" podUID="c58ed2b0-024a-48af-ab22-213400acc765" containerName="maas-api" containerID="cri-o://e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee" gracePeriod=30 Nov 03 19:36:50.865246 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.865183 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:36:50.942480 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.942448 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-metrics-certs\") pod \"c58ed2b0-024a-48af-ab22-213400acc765\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " Nov 03 19:36:50.942690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.942498 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-maas-api-tls\") pod \"c58ed2b0-024a-48af-ab22-213400acc765\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " Nov 03 19:36:50.942690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.942610 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58ed2b0-024a-48af-ab22-213400acc765-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58ed2b0-024a-48af-ab22-213400acc765-openshift-service-ca\") pod \"c58ed2b0-024a-48af-ab22-213400acc765\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " Nov 03 19:36:50.942690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.942629 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58ed2b0-024a-48af-ab22-213400acc765-kube-api-access-trfqz\" (UniqueName: \"kubernetes.io/projected/c58ed2b0-024a-48af-ab22-213400acc765-kube-api-access-trfqz\") pod \"c58ed2b0-024a-48af-ab22-213400acc765\" (UID: \"c58ed2b0-024a-48af-ab22-213400acc765\") " Nov 03 19:36:50.942996 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.942974 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58ed2b0-024a-48af-ab22-213400acc765-openshift-service-ca" pod "c58ed2b0-024a-48af-ab22-213400acc765" (UID: "c58ed2b0-024a-48af-ab22-213400acc765"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:36:50.944841 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.944814 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-metrics-certs" pod "c58ed2b0-024a-48af-ab22-213400acc765" (UID: "c58ed2b0-024a-48af-ab22-213400acc765"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:36:50.944974 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.944864 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58ed2b0-024a-48af-ab22-213400acc765-kube-api-access-trfqz" pod "c58ed2b0-024a-48af-ab22-213400acc765" (UID: "c58ed2b0-024a-48af-ab22-213400acc765"). InnerVolumeSpecName "kube-api-access-trfqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:36:50.944974 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:50.944915 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-maas-api-tls" pod "c58ed2b0-024a-48af-ab22-213400acc765" (UID: "c58ed2b0-024a-48af-ab22-213400acc765"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:36:51.043620 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.043586 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58ed2b0-024a-48af-ab22-213400acc765-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:51.043620 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.043615 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trfqz\" (UniqueName: \"kubernetes.io/projected/c58ed2b0-024a-48af-ab22-213400acc765-kube-api-access-trfqz\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:51.043620 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.043626 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:51.043872 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.043635 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58ed2b0-024a-48af-ab22-213400acc765-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:51.347702 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.347668 5019 generic.go:356] "Generic (PLEG): container finished" podID="c58ed2b0-024a-48af-ab22-213400acc765" containerID="e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee" exitCode=0 Nov 03 19:36:51.347887 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.347721 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" Nov 03 19:36:51.347887 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.347754 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" event={"ID":"c58ed2b0-024a-48af-ab22-213400acc765","Type":"ContainerDied","Data":"e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee"} Nov 03 19:36:51.347887 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.347810 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f" event={"ID":"c58ed2b0-024a-48af-ab22-213400acc765","Type":"ContainerDied","Data":"919101af7e0a89832a68c3c19e5a979b7f4a9a4006910778dd81874704062cd9"} Nov 03 19:36:51.347887 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.347834 5019 scope.go:122] "RemoveContainer" containerID="e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee" Nov 03 19:36:51.358218 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.358200 5019 scope.go:122] "RemoveContainer" containerID="e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee" Nov 03 19:36:51.358525 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:36:51.358507 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee\": container with ID starting with e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee not found: ID does not exist" containerID="e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee" Nov 03 19:36:51.358608 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.358534 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee"} err="failed to get container status \"e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee\": rpc error: code = NotFound desc = could not find container \"e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee\": container with ID starting with e0af1413a605693ee94d69e926e1a2d99af9aebaabbbac92e2ef554221963dee not found: ID does not exist" Nov 03 19:36:51.501809 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.501773 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f"] Nov 03 19:36:51.507851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:51.507824 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fcb646-6f758fc5bd-fwp9f"] Nov 03 19:36:52.149934 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:36:52.149897 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58ed2b0-024a-48af-ab22-213400acc765" path="/var/lib/kubelet/pods/c58ed2b0-024a-48af-ab22-213400acc765/volumes" Nov 03 19:37:11.296178 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:37:11.296138 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:37:19.650991 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:19.650956 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj"] Nov 03 19:37:19.651430 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:19.651182 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" podUID="d2bbf06c-ae28-42e0-8767-08283df56709" containerName="maas-api" containerID="cri-o://40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac" gracePeriod=30 Nov 03 19:37:19.900422 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:19.900390 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:37:19.941689 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:19.941656 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2bbf06c-ae28-42e0-8767-08283df56709-kube-api-access-vk2pq\" (UniqueName: \"kubernetes.io/projected/d2bbf06c-ae28-42e0-8767-08283df56709-kube-api-access-vk2pq\") pod \"d2bbf06c-ae28-42e0-8767-08283df56709\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " Nov 03 19:37:19.941852 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:19.941701 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2bbf06c-ae28-42e0-8767-08283df56709-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2bbf06c-ae28-42e0-8767-08283df56709-openshift-service-ca\") pod \"d2bbf06c-ae28-42e0-8767-08283df56709\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " Nov 03 19:37:19.941852 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:19.941722 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-metrics-certs\") pod \"d2bbf06c-ae28-42e0-8767-08283df56709\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " Nov 03 19:37:19.941852 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:19.941768 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-maas-api-tls\") pod \"d2bbf06c-ae28-42e0-8767-08283df56709\" (UID: \"d2bbf06c-ae28-42e0-8767-08283df56709\") " Nov 03 19:37:19.942114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:19.942089 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2bbf06c-ae28-42e0-8767-08283df56709-openshift-service-ca" pod "d2bbf06c-ae28-42e0-8767-08283df56709" (UID: "d2bbf06c-ae28-42e0-8767-08283df56709"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:37:19.944039 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:19.944011 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-maas-api-tls" pod "d2bbf06c-ae28-42e0-8767-08283df56709" (UID: "d2bbf06c-ae28-42e0-8767-08283df56709"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:37:19.944171 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:19.944038 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-metrics-certs" pod "d2bbf06c-ae28-42e0-8767-08283df56709" (UID: "d2bbf06c-ae28-42e0-8767-08283df56709"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:37:19.944171 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:19.944060 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bbf06c-ae28-42e0-8767-08283df56709-kube-api-access-vk2pq" pod "d2bbf06c-ae28-42e0-8767-08283df56709" (UID: "d2bbf06c-ae28-42e0-8767-08283df56709"). InnerVolumeSpecName "kube-api-access-vk2pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:20.043435 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.043400 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk2pq\" (UniqueName: \"kubernetes.io/projected/d2bbf06c-ae28-42e0-8767-08283df56709-kube-api-access-vk2pq\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:20.043435 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.043434 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2bbf06c-ae28-42e0-8767-08283df56709-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:20.043653 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.043447 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:20.043653 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.043460 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2bbf06c-ae28-42e0-8767-08283df56709-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:20.513166 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.513127 5019 generic.go:356] "Generic (PLEG): container finished" podID="d2bbf06c-ae28-42e0-8767-08283df56709" containerID="40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac" exitCode=0 Nov 03 19:37:20.513463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.513184 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" Nov 03 19:37:20.513463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.513211 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" event={"ID":"d2bbf06c-ae28-42e0-8767-08283df56709","Type":"ContainerDied","Data":"40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac"} Nov 03 19:37:20.513463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.513275 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj" event={"ID":"d2bbf06c-ae28-42e0-8767-08283df56709","Type":"ContainerDied","Data":"dde1a3d804f7dc1787147c2e8c31c726c7a91d0a054fcd4a608789c2b415d55f"} Nov 03 19:37:20.513463 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.513300 5019 scope.go:122] "RemoveContainer" containerID="40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac" Nov 03 19:37:20.522458 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.522439 5019 scope.go:122] "RemoveContainer" containerID="40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac" Nov 03 19:37:20.522739 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:37:20.522716 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac\": container with ID starting with 40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac not found: ID does not exist" containerID="40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac" Nov 03 19:37:20.522790 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.522750 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac"} err="failed to get container status \"40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac\": rpc error: code = NotFound desc = could not find container \"40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac\": container with ID starting with 40fd03d71c40fedd7f73ab6cf8f2ede9d00147eff4e81f04bbe9da332e2cd8ac not found: ID does not exist" Nov 03 19:37:20.676571 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.676539 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj"] Nov 03 19:37:20.680625 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:20.680592 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24cdad-766b9f7894-mv4xj"] Nov 03 19:37:22.150227 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:37:22.150188 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2bbf06c-ae28-42e0-8767-08283df56709" path="/var/lib/kubelet/pods/d2bbf06c-ae28-42e0-8767-08283df56709/volumes" Nov 03 19:37:41.297771 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:37:41.297682 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:38:04.262094 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.262048 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg"] Nov 03 19:38:04.262775 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.262312 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" podUID="c214f760-6b74-4f5a-bf17-5dc2fe17ed7f" containerName="maas-api" containerID="cri-o://9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a" gracePeriod=30 Nov 03 19:38:04.526117 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.525972 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 19:38:04.597080 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.597042 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-kube-api-access-7jtkf\" (UniqueName: \"kubernetes.io/projected/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-kube-api-access-7jtkf\") pod \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " Nov 03 19:38:04.597080 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.597094 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-openshift-service-ca\") pod \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " Nov 03 19:38:04.597377 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.597177 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-maas-api-tls\") pod \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " Nov 03 19:38:04.597377 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.597226 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-metrics-certs\") pod \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\" (UID: \"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f\") " Nov 03 19:38:04.597595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.597552 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-openshift-service-ca" pod "c214f760-6b74-4f5a-bf17-5dc2fe17ed7f" (UID: "c214f760-6b74-4f5a-bf17-5dc2fe17ed7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:38:04.599558 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.599527 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-kube-api-access-7jtkf" pod "c214f760-6b74-4f5a-bf17-5dc2fe17ed7f" (UID: "c214f760-6b74-4f5a-bf17-5dc2fe17ed7f"). InnerVolumeSpecName "kube-api-access-7jtkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:38:04.599797 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.599776 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-maas-api-tls" pod "c214f760-6b74-4f5a-bf17-5dc2fe17ed7f" (UID: "c214f760-6b74-4f5a-bf17-5dc2fe17ed7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:38:04.599938 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.599922 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-metrics-certs" pod "c214f760-6b74-4f5a-bf17-5dc2fe17ed7f" (UID: "c214f760-6b74-4f5a-bf17-5dc2fe17ed7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:38:04.698371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.698304 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jtkf\" (UniqueName: \"kubernetes.io/projected/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-kube-api-access-7jtkf\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:04.698371 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.698367 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:04.698603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.698382 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:04.698603 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.698394 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:04.748921 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.748885 5019 generic.go:356] "Generic (PLEG): container finished" podID="c214f760-6b74-4f5a-bf17-5dc2fe17ed7f" containerID="9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a" exitCode=0 Nov 03 19:38:04.749108 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.748930 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" event={"ID":"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f","Type":"ContainerDied","Data":"9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a"} Nov 03 19:38:04.749108 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.748971 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" event={"ID":"c214f760-6b74-4f5a-bf17-5dc2fe17ed7f","Type":"ContainerDied","Data":"dc6c8262960cfd5f3c7a7c88b36f5659f9534a7d2765943aa66391666f31e204"} Nov 03 19:38:04.749108 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.748992 5019 scope.go:122] "RemoveContainer" containerID="9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a" Nov 03 19:38:04.749108 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.748933 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg" Nov 03 19:38:04.764817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.764789 5019 scope.go:122] "RemoveContainer" containerID="9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a" Nov 03 19:38:04.765205 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:38:04.765177 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a\": container with ID starting with 9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a not found: ID does not exist" containerID="9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a" Nov 03 19:38:04.765318 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.765220 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a"} err="failed to get container status \"9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a\": rpc error: code = NotFound desc = could not find container \"9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a\": container with ID starting with 9d2b34e7102300ebade57246cf63b43ee687c7baaf6069549646832c2582706a not found: ID does not exist" Nov 03 19:38:04.902716 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.902635 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg"] Nov 03 19:38:04.906604 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:04.906570 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-619731-86b4759f9f-nnthg"] Nov 03 19:38:06.148936 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:06.148904 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c214f760-6b74-4f5a-bf17-5dc2fe17ed7f" path="/var/lib/kubelet/pods/c214f760-6b74-4f5a-bf17-5dc2fe17ed7f/volumes" Nov 03 19:38:11.296555 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:38:11.296515 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:38:19.083316 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.083281 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbcd964c6-q7xqh"] Nov 03 19:38:19.083787 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.083522 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-bbcd964c6-q7xqh" podUID="cbea2cc2-3245-4a60-8af0-1bf0292f7b62" containerName="manager" containerID="cri-o://1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6" gracePeriod=30 Nov 03 19:38:19.338622 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.338555 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 19:38:19.444462 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.444428 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-webhook-cert\") pod \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\" (UID: \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\") " Nov 03 19:38:19.444669 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.444487 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-kube-api-access-xjxd9\" (UniqueName: \"kubernetes.io/projected/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-kube-api-access-xjxd9\") pod \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\" (UID: \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\") " Nov 03 19:38:19.444669 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.444569 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-metrics-certs\") pod \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\" (UID: \"cbea2cc2-3245-4a60-8af0-1bf0292f7b62\") " Nov 03 19:38:19.446768 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.446735 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-webhook-cert" pod "cbea2cc2-3245-4a60-8af0-1bf0292f7b62" (UID: "cbea2cc2-3245-4a60-8af0-1bf0292f7b62"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:38:19.446882 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.446810 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-metrics-certs" pod "cbea2cc2-3245-4a60-8af0-1bf0292f7b62" (UID: "cbea2cc2-3245-4a60-8af0-1bf0292f7b62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:38:19.446882 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.446825 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-kube-api-access-xjxd9" pod "cbea2cc2-3245-4a60-8af0-1bf0292f7b62" (UID: "cbea2cc2-3245-4a60-8af0-1bf0292f7b62"). InnerVolumeSpecName "kube-api-access-xjxd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:38:19.545383 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.545353 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:19.545383 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.545383 5019 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-webhook-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:19.545594 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.545393 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjxd9\" (UniqueName: \"kubernetes.io/projected/cbea2cc2-3245-4a60-8af0-1bf0292f7b62-kube-api-access-xjxd9\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:19.830469 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.830433 5019 generic.go:356] "Generic (PLEG): container finished" podID="cbea2cc2-3245-4a60-8af0-1bf0292f7b62" containerID="1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6" exitCode=0 Nov 03 19:38:19.830659 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.830485 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbcd964c6-q7xqh" Nov 03 19:38:19.830659 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.830518 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbcd964c6-q7xqh" event={"ID":"cbea2cc2-3245-4a60-8af0-1bf0292f7b62","Type":"ContainerDied","Data":"1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6"} Nov 03 19:38:19.830659 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.830577 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbcd964c6-q7xqh" event={"ID":"cbea2cc2-3245-4a60-8af0-1bf0292f7b62","Type":"ContainerDied","Data":"c8eeb05f85b613e4ac04970af2521a698a85a02c38d4892ff633086653500e99"} Nov 03 19:38:19.830659 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.830606 5019 scope.go:122] "RemoveContainer" containerID="1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6" Nov 03 19:38:19.841070 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.841043 5019 scope.go:122] "RemoveContainer" containerID="1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6" Nov 03 19:38:19.841422 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:38:19.841392 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6\": container with ID starting with 1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6 not found: ID does not exist" containerID="1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6" Nov 03 19:38:19.841526 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.841429 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6"} err="failed to get container status \"1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6\": rpc error: code = NotFound desc = could not find container \"1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6\": container with ID starting with 1ee12fcdda5cb47f73bc295939749943878d4a1bc892ebdc9f952f47c7d6b0a6 not found: ID does not exist" Nov 03 19:38:19.986043 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:19.985998 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbcd964c6-q7xqh"] Nov 03 19:38:20.031215 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.031178 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bbcd964c6-q7xqh"] Nov 03 19:38:20.149641 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.149565 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbea2cc2-3245-4a60-8af0-1bf0292f7b62" path="/var/lib/kubelet/pods/cbea2cc2-3245-4a60-8af0-1bf0292f7b62/volumes" Nov 03 19:38:20.867079 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867034 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bbcd964c6-76gws"] Nov 03 19:38:20.867755 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867729 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c214f760-6b74-4f5a-bf17-5dc2fe17ed7f" containerName="maas-api" Nov 03 19:38:20.867883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867758 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c214f760-6b74-4f5a-bf17-5dc2fe17ed7f" containerName="maas-api" Nov 03 19:38:20.867883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867777 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c58ed2b0-024a-48af-ab22-213400acc765" containerName="maas-api" Nov 03 19:38:20.867883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867785 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58ed2b0-024a-48af-ab22-213400acc765" containerName="maas-api" Nov 03 19:38:20.867883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867811 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d2bbf06c-ae28-42e0-8767-08283df56709" containerName="maas-api" Nov 03 19:38:20.867883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867819 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2bbf06c-ae28-42e0-8767-08283df56709" containerName="maas-api" Nov 03 19:38:20.867883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867832 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" Nov 03 19:38:20.867883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867839 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" Nov 03 19:38:20.867883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867857 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cbea2cc2-3245-4a60-8af0-1bf0292f7b62" containerName="manager" Nov 03 19:38:20.867883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867863 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbea2cc2-3245-4a60-8af0-1bf0292f7b62" containerName="manager" Nov 03 19:38:20.867883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867876 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" containerName="istio-proxy" Nov 03 19:38:20.867883 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867884 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" containerName="istio-proxy" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867900 5019 cpu_manager.go:414] "containerMap: removing container" podUID="386109e8-f361-4d97-b76a-f3d84e85a533" containerName="istio-proxy" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867907 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386109e8-f361-4d97-b76a-f3d84e85a533" containerName="istio-proxy" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867919 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" containerName="istio-proxy" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.867926 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" containerName="istio-proxy" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.868042 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0629a8f5-b876-46b6-bb0f-5d73ae38a489" containerName="istio-proxy" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.868059 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="641da692-25ae-4e05-9e06-2417ab73918d" containerName="istio-proxy" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.868073 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c81f2ad-11c5-43a2-bde7-347f7e5f3715" containerName="istio-proxy" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.868085 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c214f760-6b74-4f5a-bf17-5dc2fe17ed7f" containerName="maas-api" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.868093 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58ed2b0-024a-48af-ab22-213400acc765" containerName="maas-api" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.868103 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2bbf06c-ae28-42e0-8767-08283df56709" containerName="maas-api" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.868114 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbea2cc2-3245-4a60-8af0-1bf0292f7b62" containerName="manager" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.868125 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdeb4f8e-b96b-4da7-a8a0-be70c56d66f8" containerName="istio-proxy" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.868135 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="386109e8-f361-4d97-b76a-f3d84e85a533" containerName="istio-proxy" Nov 03 19:38:20.868619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.868144 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3ba842-d764-4169-90fc-364512d8ebca" containerName="istio-proxy" Nov 03 19:38:20.873503 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.873467 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:20.960049 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.960007 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-webhook-cert\") pod \"maas-controller-bbcd964c6-76gws\" (UID: \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\") " pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:20.960232 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.960157 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmrt\" (UniqueName: \"kubernetes.io/projected/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-kube-api-access-vvmrt\") pod \"maas-controller-bbcd964c6-76gws\" (UID: \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\") " pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:20.960232 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.960196 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-metrics-certs\") pod \"maas-controller-bbcd964c6-76gws\" (UID: \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\") " pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:20.969079 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.969045 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 19:38:20.969388 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.969368 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 19:38:20.969517 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.969399 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hdw97\"" Nov 03 19:38:20.969517 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.969465 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 19:38:20.971321 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:20.971299 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbcd964c6-76gws"] Nov 03 19:38:21.061498 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:21.061448 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmrt\" (UniqueName: \"kubernetes.io/projected/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-kube-api-access-vvmrt\") pod \"maas-controller-bbcd964c6-76gws\" (UID: \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\") " pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:21.061714 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:21.061510 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-metrics-certs\") pod \"maas-controller-bbcd964c6-76gws\" (UID: \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\") " pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:21.061714 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:21.061553 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-webhook-cert\") pod \"maas-controller-bbcd964c6-76gws\" (UID: \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\") " pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:21.064873 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:21.064838 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-metrics-certs\") pod \"maas-controller-bbcd964c6-76gws\" (UID: \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\") " pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:21.064994 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:21.064887 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-webhook-cert\") pod \"maas-controller-bbcd964c6-76gws\" (UID: \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\") " pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:21.122092 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:21.122000 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmrt\" (UniqueName: \"kubernetes.io/projected/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-kube-api-access-vvmrt\") pod \"maas-controller-bbcd964c6-76gws\" (UID: \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\") " pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:21.192310 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:21.192265 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:21.568288 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:21.568244 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbcd964c6-76gws"] Nov 03 19:38:21.600911 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:38:21.600875 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede5a5ed_8994_4d2c_b1cf_db64f66dd984.slice/crio-fbc6e362d268190dd78534d8e23d00bf2a0c40955a746fa990a81abb46c5ea5b WatchSource:0}: Error finding container fbc6e362d268190dd78534d8e23d00bf2a0c40955a746fa990a81abb46c5ea5b: Status 404 returned error can't find the container with id fbc6e362d268190dd78534d8e23d00bf2a0c40955a746fa990a81abb46c5ea5b Nov 03 19:38:21.602899 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:21.602877 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:38:21.843649 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:21.843550 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbcd964c6-76gws" event={"ID":"ede5a5ed-8994-4d2c-b1cf-db64f66dd984","Type":"ContainerStarted","Data":"fbc6e362d268190dd78534d8e23d00bf2a0c40955a746fa990a81abb46c5ea5b"} Nov 03 19:38:22.850500 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:22.850401 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbcd964c6-76gws" event={"ID":"ede5a5ed-8994-4d2c-b1cf-db64f66dd984","Type":"ContainerStarted","Data":"f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1"} Nov 03 19:38:22.850979 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:22.850542 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:22.962770 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:22.962708 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bbcd964c6-76gws" podStartSLOduration=2.093309014 podStartE2EDuration="2.962694021s" totalImagesPullingTime="869.385007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:22.960556054 +0000 UTC m=+3236.933117243" watchObservedRunningTime="2026-11-03 19:38:22.962694021 +0000 UTC m=+3236.935255202" Nov 03 19:38:33.858036 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:33.858004 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:41.296887 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:38:41.296844 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:38:44.197701 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.197666 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr"] Nov 03 19:38:44.202916 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.202890 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.283133 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.283095 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.283133 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.283136 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.283419 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.283155 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.283419 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.283295 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.283419 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.283360 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.283419 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.283411 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.283639 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.283508 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxv2s\" (UniqueName: \"kubernetes.io/projected/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-kube-api-access-bxv2s\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.283639 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.283573 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.283784 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.283684 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.300570 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.300533 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr"] Nov 03 19:38:44.384554 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.384525 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.384728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.384584 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.384728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.384609 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.384728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.384630 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.384728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.384648 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.384728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.384681 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.384728 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.384708 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.385065 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.384747 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.385065 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.384789 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxv2s\" (UniqueName: \"kubernetes.io/projected/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-kube-api-access-bxv2s\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.385065 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.384957 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.385203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.385089 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.385203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.385125 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.385341 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.385311 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.385440 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.385421 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.387075 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.387052 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.387283 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.387266 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.441974 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.441939 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxv2s\" (UniqueName: \"kubernetes.io/projected/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-kube-api-access-bxv2s\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.442102 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.442003 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8164ba-f2ef-4586-9ee6-2d6c981c1f36-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-bkdvr\" (UID: \"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.516223 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.516148 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:44.941842 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.940423 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr"] Nov 03 19:38:44.947630 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.947588 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:38:44.947723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.947677 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:38:44.947723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.947716 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:38:44.961767 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:44.961741 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" event={"ID":"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36","Type":"ContainerStarted","Data":"03a3d390711ffba55d9b302db42c63058f4d8138a517233929250b7cbed69d87"} Nov 03 19:38:45.991620 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:45.991584 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" event={"ID":"0a8164ba-f2ef-4586-9ee6-2d6c981c1f36","Type":"ContainerStarted","Data":"e0030186854c636b70637d1d683320d319fb93e9de4f4e4e8208b5ae5990ebc9"} Nov 03 19:38:46.151696 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:46.151645 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" podStartSLOduration=2.151632542 podStartE2EDuration="2.151632542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:46.151062866 +0000 UTC m=+3260.123624047" watchObservedRunningTime="2026-11-03 19:38:46.151632542 +0000 UTC m=+3260.124193724" Nov 03 19:38:46.516533 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:46.516498 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:46.517988 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:46.517961 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bkdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:38:46.518114 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:46.518021 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" podUID="0a8164ba-f2ef-4586-9ee6-2d6c981c1f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:38:47.516740 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:47.516700 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bkdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:38:47.517136 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:47.516775 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" podUID="0a8164ba-f2ef-4586-9ee6-2d6c981c1f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:38:48.516509 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:48.516470 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bkdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:38:48.516697 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:48.516546 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" podUID="0a8164ba-f2ef-4586-9ee6-2d6c981c1f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:38:49.517031 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:49.516995 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bkdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:38:49.517479 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:49.517047 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" podUID="0a8164ba-f2ef-4586-9ee6-2d6c981c1f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:38:50.517063 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:50.517027 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bkdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:38:50.517507 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:50.517084 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" podUID="0a8164ba-f2ef-4586-9ee6-2d6c981c1f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:38:51.516938 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:51.516902 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bkdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:38:51.517138 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:51.516954 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" podUID="0a8164ba-f2ef-4586-9ee6-2d6c981c1f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:38:53.517112 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:53.517072 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bkdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:38:53.517566 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:53.517145 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" podUID="0a8164ba-f2ef-4586-9ee6-2d6c981c1f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:38:54.382248 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.382209 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbcd964c6-76gws"] Nov 03 19:38:54.382512 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.382481 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-bbcd964c6-76gws" podUID="ede5a5ed-8994-4d2c-b1cf-db64f66dd984" containerName="manager" containerID="cri-o://f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1" gracePeriod=30 Nov 03 19:38:54.558493 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.558448 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bkdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:38:54.558876 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.558510 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" podUID="0a8164ba-f2ef-4586-9ee6-2d6c981c1f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:38:54.658890 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.658868 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:54.792355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.792227 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-kube-api-access-vvmrt\" (UniqueName: \"kubernetes.io/projected/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-kube-api-access-vvmrt\") pod \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\" (UID: \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\") " Nov 03 19:38:54.792355 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.792287 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-metrics-certs\") pod \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\" (UID: \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\") " Nov 03 19:38:54.792851 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.792661 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-webhook-cert\") pod \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\" (UID: \"ede5a5ed-8994-4d2c-b1cf-db64f66dd984\") " Nov 03 19:38:54.796493 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.796374 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-kube-api-access-vvmrt" pod "ede5a5ed-8994-4d2c-b1cf-db64f66dd984" (UID: "ede5a5ed-8994-4d2c-b1cf-db64f66dd984"). InnerVolumeSpecName "kube-api-access-vvmrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:38:54.796493 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.796459 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-metrics-certs" pod "ede5a5ed-8994-4d2c-b1cf-db64f66dd984" (UID: "ede5a5ed-8994-4d2c-b1cf-db64f66dd984"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:38:54.797307 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.797282 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-webhook-cert" pod "ede5a5ed-8994-4d2c-b1cf-db64f66dd984" (UID: "ede5a5ed-8994-4d2c-b1cf-db64f66dd984"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:38:54.894676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.894600 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvmrt\" (UniqueName: \"kubernetes.io/projected/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-kube-api-access-vvmrt\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:54.894676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.894637 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:54.894676 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:54.894656 5019 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ede5a5ed-8994-4d2c-b1cf-db64f66dd984-webhook-cert\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:55.041580 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:55.041541 5019 generic.go:356] "Generic (PLEG): container finished" podID="ede5a5ed-8994-4d2c-b1cf-db64f66dd984" containerID="f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1" exitCode=0 Nov 03 19:38:55.041771 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:55.041597 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbcd964c6-76gws" Nov 03 19:38:55.041771 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:55.041620 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbcd964c6-76gws" event={"ID":"ede5a5ed-8994-4d2c-b1cf-db64f66dd984","Type":"ContainerDied","Data":"f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1"} Nov 03 19:38:55.041771 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:55.041672 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbcd964c6-76gws" event={"ID":"ede5a5ed-8994-4d2c-b1cf-db64f66dd984","Type":"ContainerDied","Data":"fbc6e362d268190dd78534d8e23d00bf2a0c40955a746fa990a81abb46c5ea5b"} Nov 03 19:38:55.041771 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:55.041694 5019 scope.go:122] "RemoveContainer" containerID="f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1" Nov 03 19:38:55.051485 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:55.051435 5019 scope.go:122] "RemoveContainer" containerID="f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1" Nov 03 19:38:55.051715 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:38:55.051697 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1\": container with ID starting with f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1 not found: ID does not exist" containerID="f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1" Nov 03 19:38:55.051767 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:55.051722 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1"} err="failed to get container status \"f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1\": rpc error: code = NotFound desc = could not find container \"f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1\": container with ID starting with f2c6ec66fe044d80bdb025f1c6f0173d7deb7b03d8998a81acc8a6f3dc2614b1 not found: ID does not exist" Nov 03 19:38:55.197183 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:55.197120 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbcd964c6-76gws"] Nov 03 19:38:55.242066 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:55.242036 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bbcd964c6-76gws"] Nov 03 19:38:55.375011 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:55.374976 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:55.375360 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:55.375315 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:55.376055 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:55.376034 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bkdvr" Nov 03 19:38:56.149381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.149350 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede5a5ed-8994-4d2c-b1cf-db64f66dd984" path="/var/lib/kubelet/pods/ede5a5ed-8994-4d2c-b1cf-db64f66dd984/volumes" Nov 03 19:38:56.591512 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.591477 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bbcd964c6-ztfsd"] Nov 03 19:38:56.591906 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.591894 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ede5a5ed-8994-4d2c-b1cf-db64f66dd984" containerName="manager" Nov 03 19:38:56.591959 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.591907 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede5a5ed-8994-4d2c-b1cf-db64f66dd984" containerName="manager" Nov 03 19:38:56.592022 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.592013 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede5a5ed-8994-4d2c-b1cf-db64f66dd984" containerName="manager" Nov 03 19:38:56.595198 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.595182 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:38:56.644081 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.644054 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 19:38:56.690338 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.690312 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hdw97\"" Nov 03 19:38:56.690467 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.690350 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 19:38:56.690467 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.690350 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 19:38:56.697191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.697168 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbcd964c6-ztfsd"] Nov 03 19:38:56.710627 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.710604 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef-webhook-cert\") pod \"maas-controller-bbcd964c6-ztfsd\" (UID: \"007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef\") " pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:38:56.710734 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.710650 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef-metrics-certs\") pod \"maas-controller-bbcd964c6-ztfsd\" (UID: \"007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef\") " pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:38:56.710818 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.710798 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4v9d\" (UniqueName: \"kubernetes.io/projected/007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef-kube-api-access-v4v9d\") pod \"maas-controller-bbcd964c6-ztfsd\" (UID: \"007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef\") " pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:38:56.811704 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.811670 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef-webhook-cert\") pod \"maas-controller-bbcd964c6-ztfsd\" (UID: \"007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef\") " pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:38:56.811853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.811717 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef-metrics-certs\") pod \"maas-controller-bbcd964c6-ztfsd\" (UID: \"007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef\") " pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:38:56.812026 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.811827 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4v9d\" (UniqueName: \"kubernetes.io/projected/007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef-kube-api-access-v4v9d\") pod \"maas-controller-bbcd964c6-ztfsd\" (UID: \"007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef\") " pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:38:56.815416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.815188 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef-metrics-certs\") pod \"maas-controller-bbcd964c6-ztfsd\" (UID: \"007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef\") " pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:38:56.815416 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.815262 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef-webhook-cert\") pod \"maas-controller-bbcd964c6-ztfsd\" (UID: \"007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef\") " pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:38:56.869954 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.869901 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4v9d\" (UniqueName: \"kubernetes.io/projected/007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef-kube-api-access-v4v9d\") pod \"maas-controller-bbcd964c6-ztfsd\" (UID: \"007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef\") " pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:38:56.908639 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:56.908611 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:38:57.276452 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:57.276420 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbcd964c6-ztfsd"] Nov 03 19:38:57.278475 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:38:57.278446 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007e5bc7_ebd6_49e0_94d1_e0a22d70d7ef.slice/crio-0dd640330c64947028bc5d6d1ca91186cd59b77382e7619e75d705027110c6ff WatchSource:0}: Error finding container 0dd640330c64947028bc5d6d1ca91186cd59b77382e7619e75d705027110c6ff: Status 404 returned error can't find the container with id 0dd640330c64947028bc5d6d1ca91186cd59b77382e7619e75d705027110c6ff Nov 03 19:38:58.059391 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:58.059359 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbcd964c6-ztfsd" event={"ID":"007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef","Type":"ContainerStarted","Data":"0dd640330c64947028bc5d6d1ca91186cd59b77382e7619e75d705027110c6ff"} Nov 03 19:38:59.065340 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:59.065291 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbcd964c6-ztfsd" event={"ID":"007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef","Type":"ContainerStarted","Data":"bbdce535a0c94b4cdea165973002403c7257428e7f80c28c58106035f67e020a"} Nov 03 19:38:59.065760 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:59.065379 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:38:59.177136 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:38:59.177083 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bbcd964c6-ztfsd" podStartSLOduration=2.358325625 podStartE2EDuration="3.177068786s" totalImagesPullingTime="818.743161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:59.175724042 +0000 UTC m=+3273.148285225" watchObservedRunningTime="2026-11-03 19:38:59.177068786 +0000 UTC m=+3273.149629997" Nov 03 19:39:10.073930 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:10.073849 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bbcd964c6-ztfsd" Nov 03 19:39:11.296687 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:39:11.296650 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:39:29.888254 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:29.888220 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:39:29.894736 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:29.894712 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:39:41.297117 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:39:41.297079 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:39:44.659195 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.659159 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8"] Nov 03 19:39:44.664486 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.664465 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.758250 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.758218 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8"] Nov 03 19:39:44.776829 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.776799 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.776980 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.776853 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab5eebd-1d3e-4390-b501-0598458c550e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.776980 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.776912 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8kp\" (UniqueName: \"kubernetes.io/projected/7ab5eebd-1d3e-4390-b501-0598458c550e-kube-api-access-zq8kp\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.776980 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.776942 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab5eebd-1d3e-4390-b501-0598458c550e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.776980 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.776963 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.777222 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.777019 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.777222 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.777151 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab5eebd-1d3e-4390-b501-0598458c550e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.777222 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.777177 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.777222 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.777196 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.877999 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.877965 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab5eebd-1d3e-4390-b501-0598458c550e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.877999 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878000 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8kp\" (UniqueName: \"kubernetes.io/projected/7ab5eebd-1d3e-4390-b501-0598458c550e-kube-api-access-zq8kp\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.878233 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878022 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab5eebd-1d3e-4390-b501-0598458c550e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.878233 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878041 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.878233 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878059 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.878233 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878125 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab5eebd-1d3e-4390-b501-0598458c550e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.878233 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878150 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.878233 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878178 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.878233 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878232 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.878690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878574 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.878690 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878643 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.878764 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878695 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.878807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878761 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.878807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.878774 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab5eebd-1d3e-4390-b501-0598458c550e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.880481 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.880460 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab5eebd-1d3e-4390-b501-0598458c550e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.880710 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.880693 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab5eebd-1d3e-4390-b501-0598458c550e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.940412 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.940390 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab5eebd-1d3e-4390-b501-0598458c550e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.940720 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.940697 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8kp\" (UniqueName: \"kubernetes.io/projected/7ab5eebd-1d3e-4390-b501-0598458c550e-kube-api-access-zq8kp\") pod \"maas-default-gateway-openshift-default-77f556d647-4hpg8\" (UID: \"7ab5eebd-1d3e-4390-b501-0598458c550e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:44.977616 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:44.977591 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:45.394120 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:39:45.394086 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab5eebd_1d3e_4390_b501_0598458c550e.slice/crio-e249d3dd02133767a8c2f233ceb1ba2399442a155b3de7020acc53b31009dca1 WatchSource:0}: Error finding container e249d3dd02133767a8c2f233ceb1ba2399442a155b3de7020acc53b31009dca1: Status 404 returned error can't find the container with id e249d3dd02133767a8c2f233ceb1ba2399442a155b3de7020acc53b31009dca1 Nov 03 19:39:45.396419 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:45.396380 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:39:45.396526 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:45.396455 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:39:45.396526 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:45.396487 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:39:45.397884 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:45.397857 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8"] Nov 03 19:39:46.312965 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:46.312933 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" event={"ID":"7ab5eebd-1d3e-4390-b501-0598458c550e","Type":"ContainerStarted","Data":"c1b4f38484a1a28dfa8cdee18d9894afdb1a7ff3eab88d7a1f1179d599b8f3ba"} Nov 03 19:39:46.313392 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:46.312984 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" event={"ID":"7ab5eebd-1d3e-4390-b501-0598458c550e","Type":"ContainerStarted","Data":"e249d3dd02133767a8c2f233ceb1ba2399442a155b3de7020acc53b31009dca1"} Nov 03 19:39:46.437392 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:46.437288 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" podStartSLOduration=2.437274785 podStartE2EDuration="2.437274785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:46.436895997 +0000 UTC m=+3320.409457161" watchObservedRunningTime="2026-11-03 19:39:46.437274785 +0000 UTC m=+3320.409835966" Nov 03 19:39:46.980879 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:46.980219 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:46.981199 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:46.981178 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4hpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 19:39:46.981390 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:46.981365 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" podUID="7ab5eebd-1d3e-4390-b501-0598458c550e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 19:39:47.983742 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:47.983702 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4hpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 19:39:47.984370 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:47.984312 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" podUID="7ab5eebd-1d3e-4390-b501-0598458c550e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 19:39:48.980351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:48.978700 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4hpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 19:39:48.980351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:48.978761 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" podUID="7ab5eebd-1d3e-4390-b501-0598458c550e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 19:39:49.981352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:49.980390 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4hpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 19:39:49.981352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:49.980448 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" podUID="7ab5eebd-1d3e-4390-b501-0598458c550e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 19:39:50.985352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:50.983576 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4hpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 19:39:50.985352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:50.983659 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" podUID="7ab5eebd-1d3e-4390-b501-0598458c550e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 19:39:51.977976 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:51.977927 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4hpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 19:39:51.978196 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:51.977998 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" podUID="7ab5eebd-1d3e-4390-b501-0598458c550e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 19:39:53.979393 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:53.979349 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4hpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:39:53.979879 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:53.979418 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" podUID="7ab5eebd-1d3e-4390-b501-0598458c550e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:39:55.021524 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:55.021485 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4hpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:39:55.022030 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:55.021565 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" podUID="7ab5eebd-1d3e-4390-b501-0598458c550e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:39:55.730817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:55.730782 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:55.731205 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:55.731184 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:39:55.731875 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:39:55.731858 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4hpg8" Nov 03 19:40:11.296077 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:40:11.296034 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:40:14.883691 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:14.883660 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw"] Nov 03 19:40:14.887659 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:14.887640 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:14.996707 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:14.996672 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw"] Nov 03 19:40:14.999891 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:14.999868 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb93eda2-29b3-439e-a19d-984ecad5177f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:14.999952 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:14.999909 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:14.999952 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:14.999936 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.000067 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.000049 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb93eda2-29b3-439e-a19d-984ecad5177f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.000104 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.000083 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.000138 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.000109 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.000172 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.000143 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb93eda2-29b3-439e-a19d-984ecad5177f-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.000259 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.000233 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.000349 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.000312 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9l8t\" (UniqueName: \"kubernetes.io/projected/eb93eda2-29b3-439e-a19d-984ecad5177f-kube-api-access-k9l8t\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.101515 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.101463 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.101515 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.101530 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.101787 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.101554 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb93eda2-29b3-439e-a19d-984ecad5177f-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.101787 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.101578 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.101787 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.101613 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9l8t\" (UniqueName: \"kubernetes.io/projected/eb93eda2-29b3-439e-a19d-984ecad5177f-kube-api-access-k9l8t\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.101787 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.101666 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb93eda2-29b3-439e-a19d-984ecad5177f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.101787 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.101708 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.101787 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.101748 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.102088 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.101805 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb93eda2-29b3-439e-a19d-984ecad5177f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.102088 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.101912 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.102088 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.101973 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.102088 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.102065 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.102220 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.102169 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.102555 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.102533 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb93eda2-29b3-439e-a19d-984ecad5177f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.103978 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.103958 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb93eda2-29b3-439e-a19d-984ecad5177f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.104352 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.104318 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb93eda2-29b3-439e-a19d-984ecad5177f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.164106 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.164032 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb93eda2-29b3-439e-a19d-984ecad5177f-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.164106 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.164090 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9l8t\" (UniqueName: \"kubernetes.io/projected/eb93eda2-29b3-439e-a19d-984ecad5177f-kube-api-access-k9l8t\") pod \"maas-default-gateway-openshift-default-77f556d647-75zmw\" (UID: \"eb93eda2-29b3-439e-a19d-984ecad5177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.200983 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.200952 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:15.615491 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:40:15.615462 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb93eda2_29b3_439e_a19d_984ecad5177f.slice/crio-8049bc2a9bba134443e246a4e53459b8d0431f595bf95c259f37fba1b41aa7ee WatchSource:0}: Error finding container 8049bc2a9bba134443e246a4e53459b8d0431f595bf95c259f37fba1b41aa7ee: Status 404 returned error can't find the container with id 8049bc2a9bba134443e246a4e53459b8d0431f595bf95c259f37fba1b41aa7ee Nov 03 19:40:15.617926 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.617880 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:40:15.618029 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.617971 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:40:15.618029 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.618014 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:40:15.619205 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:15.619182 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw"] Nov 03 19:40:16.491777 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:16.491744 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" event={"ID":"eb93eda2-29b3-439e-a19d-984ecad5177f","Type":"ContainerStarted","Data":"71be89dbe0a2a27ebd2669d08ab73d488ff5c159284a838d824740f0195f88e2"} Nov 03 19:40:16.492206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:16.491798 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" event={"ID":"eb93eda2-29b3-439e-a19d-984ecad5177f","Type":"ContainerStarted","Data":"8049bc2a9bba134443e246a4e53459b8d0431f595bf95c259f37fba1b41aa7ee"} Nov 03 19:40:17.201954 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:17.201911 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:17.203723 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:17.203696 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-75zmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 19:40:17.203849 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:17.203743 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" podUID="eb93eda2-29b3-439e-a19d-984ecad5177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 19:40:18.201889 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:18.201858 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-75zmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 19:40:18.202295 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:18.201911 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" podUID="eb93eda2-29b3-439e-a19d-984ecad5177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 19:40:19.201937 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:19.201898 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-75zmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 19:40:19.202465 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:19.201961 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" podUID="eb93eda2-29b3-439e-a19d-984ecad5177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 19:40:20.202287 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:20.202251 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-75zmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 19:40:20.202807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:20.202304 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" podUID="eb93eda2-29b3-439e-a19d-984ecad5177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 19:40:21.202415 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:21.202379 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-75zmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 19:40:21.202819 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:21.202433 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" podUID="eb93eda2-29b3-439e-a19d-984ecad5177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 19:40:22.201633 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:22.201599 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-75zmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 19:40:22.201813 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:22.201651 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" podUID="eb93eda2-29b3-439e-a19d-984ecad5177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 19:40:23.803175 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:23.803139 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:23.803741 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:23.803714 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:23.804624 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:23.804603 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" Nov 03 19:40:23.979404 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:40:23.979294 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" podStartSLOduration=9.979280324 podStartE2EDuration="9.979280324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:16.685724785 +0000 UTC m=+3350.658285968" watchObservedRunningTime="2026-11-03 19:40:23.979280324 +0000 UTC m=+3357.951841578" Nov 03 19:40:41.297420 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:40:41.297315 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:41:11.296418 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:41:11.296385 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:41:41.295894 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:41:41.295859 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:42:11.295908 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:42:11.295828 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:42:41.295812 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:42:41.295772 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:43:11.296900 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:43:11.296855 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:43:41.295858 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:43:41.295771 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:44:11.295481 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:44:11.295447 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:44:22.459875 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.459839 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl"] Nov 03 19:44:22.463847 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.463827 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:22.517914 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.517885 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-8ecb68-serving-cert\"" Nov 03 19:44:22.571541 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.571506 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl"] Nov 03 19:44:22.608790 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.608754 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:22.608974 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.608821 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:22.608974 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.608969 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rvkb\" (UniqueName: \"kubernetes.io/projected/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-kube-api-access-6rvkb\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:22.609098 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.609018 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:22.711436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.711290 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:22.714919 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.714870 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:22.715833 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.715805 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:22.716200 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.716177 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rvkb\" (UniqueName: \"kubernetes.io/projected/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-kube-api-access-6rvkb\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:22.716771 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:44:22.716098 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-serving-cert: secret "maas-api-e2e-worker-single-main-8ecb68-serving-cert" not found Nov 03 19:44:22.716959 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:44:22.716948 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-maas-api-tls podName:39c9a3da-f6da-468a-acd9-0ec8a2be1cb5 nodeName:}" failed. No retries permitted until 2026-11-03 19:44:23.216923311 +0000 UTC m=+3597.189484493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-maas-api-tls") pod "maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" (UID: "39c9a3da-f6da-468a-acd9-0ec8a2be1cb5") : secret "maas-api-e2e-worker-single-main-8ecb68-serving-cert" not found Nov 03 19:44:22.727351 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.727118 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:22.742658 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.742614 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:22.781033 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:22.781007 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rvkb\" (UniqueName: \"kubernetes.io/projected/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-kube-api-access-6rvkb\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:23.221980 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:23.221943 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:23.224522 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:23.224498 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:23.376512 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:23.376470 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:23.747300 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:44:23.747271 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c9a3da_f6da_468a_acd9_0ec8a2be1cb5.slice/crio-5d9f9f78d7b37a1e3bc68ba38318ab7b3f6558be307d511b47e6347a323f2cb2 WatchSource:0}: Error finding container 5d9f9f78d7b37a1e3bc68ba38318ab7b3f6558be307d511b47e6347a323f2cb2: Status 404 returned error can't find the container with id 5d9f9f78d7b37a1e3bc68ba38318ab7b3f6558be307d511b47e6347a323f2cb2 Nov 03 19:44:23.749296 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:23.749275 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:44:23.750569 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:23.750549 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl"] Nov 03 19:44:24.035224 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:24.035140 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" event={"ID":"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5","Type":"ContainerStarted","Data":"5d9f9f78d7b37a1e3bc68ba38318ab7b3f6558be307d511b47e6347a323f2cb2"} Nov 03 19:44:25.042242 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:25.042152 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" event={"ID":"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5","Type":"ContainerStarted","Data":"c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886"} Nov 03 19:44:25.042242 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:25.042231 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:25.162575 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:25.162528 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" podStartSLOduration=2.193358235 podStartE2EDuration="3.16251549s" totalImagesPullingTime="969.157255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:25.162219308 +0000 UTC m=+3599.134780490" watchObservedRunningTime="2026-11-03 19:44:25.16251549 +0000 UTC m=+3599.135076672" Nov 03 19:44:29.923040 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:29.923013 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:44:29.930931 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:29.930910 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:44:31.071203 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:44:31.071173 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:44:41.298903 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:44:41.298862 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:45:11.296062 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:45:11.295979 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:45:41.297024 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:45:41.296987 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:45:52.800807 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:52.800765 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl"] Nov 03 19:45:52.801301 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:52.800983 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" podUID="39c9a3da-f6da-468a-acd9-0ec8a2be1cb5" containerName="maas-api" containerID="cri-o://c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886" gracePeriod=30 Nov 03 19:45:54.063020 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.062985 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:45:54.116367 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.116319 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-kube-api-access-6rvkb\" (UniqueName: \"kubernetes.io/projected/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-kube-api-access-6rvkb\") pod \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " Nov 03 19:45:54.116551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.116450 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-openshift-service-ca\") pod \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " Nov 03 19:45:54.116551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.116493 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-maas-api-tls\") pod \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " Nov 03 19:45:54.116551 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.116528 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-metrics-certs\") pod \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\" (UID: \"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5\") " Nov 03 19:45:54.116855 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.116821 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-openshift-service-ca" pod "39c9a3da-f6da-468a-acd9-0ec8a2be1cb5" (UID: "39c9a3da-f6da-468a-acd9-0ec8a2be1cb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:54.118619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.118590 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-metrics-certs" pod "39c9a3da-f6da-468a-acd9-0ec8a2be1cb5" (UID: "39c9a3da-f6da-468a-acd9-0ec8a2be1cb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:45:54.118763 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.118668 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-maas-api-tls" pod "39c9a3da-f6da-468a-acd9-0ec8a2be1cb5" (UID: "39c9a3da-f6da-468a-acd9-0ec8a2be1cb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:45:54.118835 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.118800 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-kube-api-access-6rvkb" pod "39c9a3da-f6da-468a-acd9-0ec8a2be1cb5" (UID: "39c9a3da-f6da-468a-acd9-0ec8a2be1cb5"). InnerVolumeSpecName "kube-api-access-6rvkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:54.217838 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.217802 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:54.217838 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.217834 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:54.218036 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.217847 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:54.218036 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.217860 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rvkb\" (UniqueName: \"kubernetes.io/projected/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5-kube-api-access-6rvkb\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:54.542517 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.542476 5019 generic.go:356] "Generic (PLEG): container finished" podID="39c9a3da-f6da-468a-acd9-0ec8a2be1cb5" containerID="c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886" exitCode=0 Nov 03 19:45:54.542694 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.542526 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" Nov 03 19:45:54.542694 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.542562 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" event={"ID":"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5","Type":"ContainerDied","Data":"c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886"} Nov 03 19:45:54.542694 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.542618 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl" event={"ID":"39c9a3da-f6da-468a-acd9-0ec8a2be1cb5","Type":"ContainerDied","Data":"5d9f9f78d7b37a1e3bc68ba38318ab7b3f6558be307d511b47e6347a323f2cb2"} Nov 03 19:45:54.542694 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.542641 5019 scope.go:122] "RemoveContainer" containerID="c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886" Nov 03 19:45:54.552155 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.552133 5019 scope.go:122] "RemoveContainer" containerID="c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886" Nov 03 19:45:54.552419 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:45:54.552402 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886\": container with ID starting with c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886 not found: ID does not exist" containerID="c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886" Nov 03 19:45:54.552476 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.552430 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886"} err="failed to get container status \"c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886\": rpc error: code = NotFound desc = could not find container \"c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886\": container with ID starting with c7d70e04538e3516f1f3b53b7fe74e09e222cbe23ee1752423d0ea801e384886 not found: ID does not exist" Nov 03 19:45:54.711574 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.711541 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl"] Nov 03 19:45:54.715120 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:54.715098 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8ecb68-5b445dcdb5-cjxfl"] Nov 03 19:45:56.150206 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:45:56.150174 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c9a3da-f6da-468a-acd9-0ec8a2be1cb5" path="/var/lib/kubelet/pods/39c9a3da-f6da-468a-acd9-0ec8a2be1cb5/volumes" Nov 03 19:46:11.295302 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:46:11.295258 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:46:22.876858 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:22.876824 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr"] Nov 03 19:46:22.877446 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:22.877042 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" podUID="2ea9cb01-6cb5-45bf-bf93-2c00a3283b58" containerName="manager" containerID="cri-o://c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514" gracePeriod=10 Nov 03 19:46:23.146973 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.146949 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" Nov 03 19:46:23.311180 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.311143 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-extensions-socket-volume\") pod \"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58\" (UID: \"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58\") " Nov 03 19:46:23.311384 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.311257 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-kube-api-access-lsl6t\" (UniqueName: \"kubernetes.io/projected/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-kube-api-access-lsl6t\") pod \"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58\" (UID: \"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58\") " Nov 03 19:46:23.311537 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.311514 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-extensions-socket-volume" pod "2ea9cb01-6cb5-45bf-bf93-2c00a3283b58" (UID: "2ea9cb01-6cb5-45bf-bf93-2c00a3283b58"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:46:23.311751 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.311734 5019 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-extensions-socket-volume\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:46:23.313278 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.313262 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-kube-api-access-lsl6t" pod "2ea9cb01-6cb5-45bf-bf93-2c00a3283b58" (UID: "2ea9cb01-6cb5-45bf-bf93-2c00a3283b58"). InnerVolumeSpecName "kube-api-access-lsl6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:46:23.413391 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.413276 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsl6t\" (UniqueName: \"kubernetes.io/projected/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58-kube-api-access-lsl6t\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:46:23.692735 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.692702 5019 generic.go:356] "Generic (PLEG): container finished" podID="2ea9cb01-6cb5-45bf-bf93-2c00a3283b58" containerID="c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514" exitCode=0 Nov 03 19:46:23.692901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.692764 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" Nov 03 19:46:23.692901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.692781 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" event={"ID":"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58","Type":"ContainerDied","Data":"c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514"} Nov 03 19:46:23.692901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.692863 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr" event={"ID":"2ea9cb01-6cb5-45bf-bf93-2c00a3283b58","Type":"ContainerDied","Data":"b929d88eaa9123a2e3d2168a73b63aa39e2444b1b9a8b48f50952342137312ba"} Nov 03 19:46:23.692901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.692886 5019 scope.go:122] "RemoveContainer" containerID="c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514" Nov 03 19:46:23.704296 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.704277 5019 scope.go:122] "RemoveContainer" containerID="c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514" Nov 03 19:46:23.704600 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:46:23.704575 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514\": container with ID starting with c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514 not found: ID does not exist" containerID="c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514" Nov 03 19:46:23.704707 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.704605 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514"} err="failed to get container status \"c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514\": rpc error: code = NotFound desc = could not find container \"c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514\": container with ID starting with c94c176169fe5191b6ffeeb0b12b2b4a3f897c58ea8f177d99ba2b5692d6d514 not found: ID does not exist" Nov 03 19:46:23.862765 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.862730 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr"] Nov 03 19:46:23.914198 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:23.914171 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xhdlr"] Nov 03 19:46:24.150309 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:46:24.150232 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea9cb01-6cb5-45bf-bf93-2c00a3283b58" path="/var/lib/kubelet/pods/2ea9cb01-6cb5-45bf-bf93-2c00a3283b58/volumes" Nov 03 19:46:41.296381 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:46:41.296279 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:47:11.296424 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:47:11.296390 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:47:26.452863 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.452826 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj"] Nov 03 19:47:26.453313 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.453300 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2ea9cb01-6cb5-45bf-bf93-2c00a3283b58" containerName="manager" Nov 03 19:47:26.453420 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.453313 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea9cb01-6cb5-45bf-bf93-2c00a3283b58" containerName="manager" Nov 03 19:47:26.453420 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.453342 5019 cpu_manager.go:414] "containerMap: removing container" podUID="39c9a3da-f6da-468a-acd9-0ec8a2be1cb5" containerName="maas-api" Nov 03 19:47:26.453420 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.453348 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c9a3da-f6da-468a-acd9-0ec8a2be1cb5" containerName="maas-api" Nov 03 19:47:26.453528 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.453425 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c9a3da-f6da-468a-acd9-0ec8a2be1cb5" containerName="maas-api" Nov 03 19:47:26.453528 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.453444 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea9cb01-6cb5-45bf-bf93-2c00a3283b58" containerName="manager" Nov 03 19:47:26.456652 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.456636 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" Nov 03 19:47:26.503692 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.503662 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj"] Nov 03 19:47:26.530198 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.530167 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8cnr\" (UniqueName: \"kubernetes.io/projected/11b09062-eabd-4950-875a-a0fd97ab66ee-kube-api-access-d8cnr\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p5zkj\" (UID: \"11b09062-eabd-4950-875a-a0fd97ab66ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" Nov 03 19:47:26.530381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.530356 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11b09062-eabd-4950-875a-a0fd97ab66ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p5zkj\" (UID: \"11b09062-eabd-4950-875a-a0fd97ab66ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" Nov 03 19:47:26.552822 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.552795 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-b2nmw\"" Nov 03 19:47:26.631301 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.631265 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11b09062-eabd-4950-875a-a0fd97ab66ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p5zkj\" (UID: \"11b09062-eabd-4950-875a-a0fd97ab66ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" Nov 03 19:47:26.631528 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.631337 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8cnr\" (UniqueName: \"kubernetes.io/projected/11b09062-eabd-4950-875a-a0fd97ab66ee-kube-api-access-d8cnr\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p5zkj\" (UID: \"11b09062-eabd-4950-875a-a0fd97ab66ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" Nov 03 19:47:26.631671 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.631649 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11b09062-eabd-4950-875a-a0fd97ab66ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p5zkj\" (UID: \"11b09062-eabd-4950-875a-a0fd97ab66ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" Nov 03 19:47:26.696109 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.696078 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8cnr\" (UniqueName: \"kubernetes.io/projected/11b09062-eabd-4950-875a-a0fd97ab66ee-kube-api-access-d8cnr\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p5zkj\" (UID: \"11b09062-eabd-4950-875a-a0fd97ab66ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" Nov 03 19:47:26.769518 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:26.769440 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" Nov 03 19:47:27.186461 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:47:27.186428 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b09062_eabd_4950_875a_a0fd97ab66ee.slice/crio-33cc9d86ada2fa6e263cd9f0cfd20816d1852ce6020e74814387d2aa0a1df292 WatchSource:0}: Error finding container 33cc9d86ada2fa6e263cd9f0cfd20816d1852ce6020e74814387d2aa0a1df292: Status 404 returned error can't find the container with id 33cc9d86ada2fa6e263cd9f0cfd20816d1852ce6020e74814387d2aa0a1df292 Nov 03 19:47:27.190397 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:27.190098 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj"] Nov 03 19:47:28.008460 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:28.008417 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" event={"ID":"11b09062-eabd-4950-875a-a0fd97ab66ee","Type":"ContainerStarted","Data":"af8998e5baf401cfde7a5d79a900d6714a106ff57f0737e49c73823d316c432a"} Nov 03 19:47:28.008853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:28.008481 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" event={"ID":"11b09062-eabd-4950-875a-a0fd97ab66ee","Type":"ContainerStarted","Data":"33cc9d86ada2fa6e263cd9f0cfd20816d1852ce6020e74814387d2aa0a1df292"} Nov 03 19:47:28.008853 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:28.008656 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" Nov 03 19:47:28.233193 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:28.233134 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" podStartSLOduration=2.233117286 podStartE2EDuration="2.233117286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:28.181048486 +0000 UTC m=+3782.153609667" watchObservedRunningTime="2026-11-03 19:47:28.233117286 +0000 UTC m=+3782.205678468" Nov 03 19:47:39.016257 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:47:39.016227 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5zkj" Nov 03 19:47:41.295722 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:47:41.295681 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:48:11.297875 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:48:11.297785 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:48:40.063235 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.063200 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml"] Nov 03 19:48:40.067191 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.067174 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.123867 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.123827 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml"] Nov 03 19:48:40.171470 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.171437 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ccf773-serving-cert\"" Nov 03 19:48:40.204122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.204093 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.204292 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.204131 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.204292 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.204210 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwp6\" (UniqueName: \"kubernetes.io/projected/36e3f540-6668-494f-b811-ad34f203f4f3-kube-api-access-2fwp6\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.204446 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.204376 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e3f540-6668-494f-b811-ad34f203f4f3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.305825 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.305785 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.306012 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:48:40.305960 5019 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-serving-cert: secret "maas-api-e2e-worker-single-main-ccf773-serving-cert" not found Nov 03 19:48:40.306070 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:48:40.306015 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-maas-api-tls podName:36e3f540-6668-494f-b811-ad34f203f4f3 nodeName:}" failed. No retries permitted until 2026-11-03 19:48:40.805998353 +0000 UTC m=+3854.778559512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-maas-api-tls") pod "maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" (UID: "36e3f540-6668-494f-b811-ad34f203f4f3") : secret "maas-api-e2e-worker-single-main-ccf773-serving-cert" not found Nov 03 19:48:40.306070 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.306043 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.306161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.306091 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwp6\" (UniqueName: \"kubernetes.io/projected/36e3f540-6668-494f-b811-ad34f203f4f3-kube-api-access-2fwp6\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.306161 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.306155 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e3f540-6668-494f-b811-ad34f203f4f3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.306766 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.306744 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e3f540-6668-494f-b811-ad34f203f4f3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.308402 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.308374 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.370800 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.370728 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwp6\" (UniqueName: \"kubernetes.io/projected/36e3f540-6668-494f-b811-ad34f203f4f3-kube-api-access-2fwp6\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.819499 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.819464 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.821818 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.821791 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:40.980694 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:40.980659 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:41.296708 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:48:41.296606 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:48:41.348290 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:48:41.348262 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e3f540_6668_494f_b811_ad34f203f4f3.slice/crio-9b8c2b69d29d5e2e05a3925927f9e0b6332f171f3fadc6ab39f61cff4f7c77f6 WatchSource:0}: Error finding container 9b8c2b69d29d5e2e05a3925927f9e0b6332f171f3fadc6ab39f61cff4f7c77f6: Status 404 returned error can't find the container with id 9b8c2b69d29d5e2e05a3925927f9e0b6332f171f3fadc6ab39f61cff4f7c77f6 Nov 03 19:48:41.351802 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:41.351771 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml"] Nov 03 19:48:41.447033 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:41.447001 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" event={"ID":"36e3f540-6668-494f-b811-ad34f203f4f3","Type":"ContainerStarted","Data":"9b8c2b69d29d5e2e05a3925927f9e0b6332f171f3fadc6ab39f61cff4f7c77f6"} Nov 03 19:48:42.454184 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:42.454149 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" event={"ID":"36e3f540-6668-494f-b811-ad34f203f4f3","Type":"ContainerStarted","Data":"e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1"} Nov 03 19:48:42.454717 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:42.454275 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:48:42.576072 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:42.575957 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" podStartSLOduration=1.6972288089999998 podStartE2EDuration="2.575939238s" totalImagesPullingTime="878.710429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:42.575017782 +0000 UTC m=+3856.547578963" watchObservedRunningTime="2026-11-03 19:48:42.575939238 +0000 UTC m=+3856.548500420" Nov 03 19:48:48.465647 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:48:48.465618 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:49:11.296878 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:49:11.296841 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:49:29.955078 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:49:29.955046 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:49:29.964821 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:49:29.964799 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:49:41.296563 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:49:41.296487 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:50:01.984034 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:01.984002 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml"] Nov 03 19:50:01.984731 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:01.984218 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" podUID="36e3f540-6668-494f-b811-ad34f203f4f3" containerName="maas-api" containerID="cri-o://e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1" gracePeriod=30 Nov 03 19:50:03.251155 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.251125 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:50:03.275255 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.275223 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-metrics-certs\") pod \"36e3f540-6668-494f-b811-ad34f203f4f3\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " Nov 03 19:50:03.275470 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.275259 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-maas-api-tls\") pod \"36e3f540-6668-494f-b811-ad34f203f4f3\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " Nov 03 19:50:03.275470 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.275288 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e3f540-6668-494f-b811-ad34f203f4f3-kube-api-access-2fwp6\" (UniqueName: \"kubernetes.io/projected/36e3f540-6668-494f-b811-ad34f203f4f3-kube-api-access-2fwp6\") pod \"36e3f540-6668-494f-b811-ad34f203f4f3\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " Nov 03 19:50:03.275470 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.275344 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36e3f540-6668-494f-b811-ad34f203f4f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e3f540-6668-494f-b811-ad34f203f4f3-openshift-service-ca\") pod \"36e3f540-6668-494f-b811-ad34f203f4f3\" (UID: \"36e3f540-6668-494f-b811-ad34f203f4f3\") " Nov 03 19:50:03.276297 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.275802 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36e3f540-6668-494f-b811-ad34f203f4f3-openshift-service-ca" pod "36e3f540-6668-494f-b811-ad34f203f4f3" (UID: "36e3f540-6668-494f-b811-ad34f203f4f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:03.277695 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.277668 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-maas-api-tls" pod "36e3f540-6668-494f-b811-ad34f203f4f3" (UID: "36e3f540-6668-494f-b811-ad34f203f4f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:03.277799 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.277700 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-metrics-certs" pod "36e3f540-6668-494f-b811-ad34f203f4f3" (UID: "36e3f540-6668-494f-b811-ad34f203f4f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:03.277799 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.277787 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e3f540-6668-494f-b811-ad34f203f4f3-kube-api-access-2fwp6" pod "36e3f540-6668-494f-b811-ad34f203f4f3" (UID: "36e3f540-6668-494f-b811-ad34f203f4f3"). InnerVolumeSpecName "kube-api-access-2fwp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:03.376750 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.376666 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:03.376750 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.376705 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36e3f540-6668-494f-b811-ad34f203f4f3-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:03.376750 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.376722 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fwp6\" (UniqueName: \"kubernetes.io/projected/36e3f540-6668-494f-b811-ad34f203f4f3-kube-api-access-2fwp6\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:03.376750 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.376740 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e3f540-6668-494f-b811-ad34f203f4f3-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:03.876097 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.876057 5019 generic.go:356] "Generic (PLEG): container finished" podID="36e3f540-6668-494f-b811-ad34f203f4f3" containerID="e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1" exitCode=0 Nov 03 19:50:03.876312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.876109 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" event={"ID":"36e3f540-6668-494f-b811-ad34f203f4f3","Type":"ContainerDied","Data":"e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1"} Nov 03 19:50:03.876312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.876110 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" Nov 03 19:50:03.876312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.876150 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml" event={"ID":"36e3f540-6668-494f-b811-ad34f203f4f3","Type":"ContainerDied","Data":"9b8c2b69d29d5e2e05a3925927f9e0b6332f171f3fadc6ab39f61cff4f7c77f6"} Nov 03 19:50:03.876312 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.876173 5019 scope.go:122] "RemoveContainer" containerID="e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1" Nov 03 19:50:03.886381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.886355 5019 scope.go:122] "RemoveContainer" containerID="e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1" Nov 03 19:50:03.886671 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:50:03.886650 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1\": container with ID starting with e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1 not found: ID does not exist" containerID="e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1" Nov 03 19:50:03.886727 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:03.886680 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1"} err="failed to get container status \"e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1\": rpc error: code = NotFound desc = could not find container \"e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1\": container with ID starting with e8e425d419834335cdbddf357ef0b972e5a8b643b37fe6aee03f84b8003b2ec1 not found: ID does not exist" Nov 03 19:50:04.045458 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:04.045422 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml"] Nov 03 19:50:04.049063 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:04.049034 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ccf773-75789df474-lbfml"] Nov 03 19:50:04.150526 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:04.150441 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e3f540-6668-494f-b811-ad34f203f4f3" path="/var/lib/kubelet/pods/36e3f540-6668-494f-b811-ad34f203f4f3/volumes" Nov 03 19:50:11.296415 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:50:11.296377 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:50:24.853496 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:24.853453 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-75zmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:50:24.854211 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:50:24.854176 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-75zmw" podUID="eb93eda2-29b3-439e-a19d-984ecad5177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:41.297052 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:50:41.297014 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:51:11.297469 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:51:11.297361 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:51:41.296755 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:51:41.296721 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:52:11.295658 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:52:11.295625 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:52:41.296461 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:52:41.296378 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:52:48.746885 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.746851 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-76fvf"] Nov 03 19:52:48.747461 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.747363 5019 cpu_manager.go:414] "containerMap: removing container" podUID="36e3f540-6668-494f-b811-ad34f203f4f3" containerName="maas-api" Nov 03 19:52:48.747461 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.747376 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e3f540-6668-494f-b811-ad34f203f4f3" containerName="maas-api" Nov 03 19:52:48.747461 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.747445 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e3f540-6668-494f-b811-ad34f203f4f3" containerName="maas-api" Nov 03 19:52:48.750817 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.750796 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:48.850778 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.850745 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjq8\" (UniqueName: \"kubernetes.io/projected/94d86bba-44b6-4065-8006-575e52434733-kube-api-access-kzjq8\") pod \"maas-api-595cc648c-76fvf\" (UID: \"94d86bba-44b6-4065-8006-575e52434733\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:48.850943 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.850822 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d86bba-44b6-4065-8006-575e52434733-metrics-certs\") pod \"maas-api-595cc648c-76fvf\" (UID: \"94d86bba-44b6-4065-8006-575e52434733\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:48.850943 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.850912 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d86bba-44b6-4065-8006-575e52434733-openshift-service-ca\") pod \"maas-api-595cc648c-76fvf\" (UID: \"94d86bba-44b6-4065-8006-575e52434733\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:48.851037 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.851016 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d86bba-44b6-4065-8006-575e52434733-maas-api-tls\") pod \"maas-api-595cc648c-76fvf\" (UID: \"94d86bba-44b6-4065-8006-575e52434733\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:48.857910 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.857882 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-76fvf"] Nov 03 19:52:48.952545 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.952510 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjq8\" (UniqueName: \"kubernetes.io/projected/94d86bba-44b6-4065-8006-575e52434733-kube-api-access-kzjq8\") pod \"maas-api-595cc648c-76fvf\" (UID: \"94d86bba-44b6-4065-8006-575e52434733\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:48.952727 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.952570 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d86bba-44b6-4065-8006-575e52434733-metrics-certs\") pod \"maas-api-595cc648c-76fvf\" (UID: \"94d86bba-44b6-4065-8006-575e52434733\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:48.952727 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.952597 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d86bba-44b6-4065-8006-575e52434733-openshift-service-ca\") pod \"maas-api-595cc648c-76fvf\" (UID: \"94d86bba-44b6-4065-8006-575e52434733\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:48.952727 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.952646 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d86bba-44b6-4065-8006-575e52434733-maas-api-tls\") pod \"maas-api-595cc648c-76fvf\" (UID: \"94d86bba-44b6-4065-8006-575e52434733\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:48.953440 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.953411 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d86bba-44b6-4065-8006-575e52434733-openshift-service-ca\") pod \"maas-api-595cc648c-76fvf\" (UID: \"94d86bba-44b6-4065-8006-575e52434733\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:48.955130 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.955108 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d86bba-44b6-4065-8006-575e52434733-metrics-certs\") pod \"maas-api-595cc648c-76fvf\" (UID: \"94d86bba-44b6-4065-8006-575e52434733\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:48.955217 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:48.955169 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d86bba-44b6-4065-8006-575e52434733-maas-api-tls\") pod \"maas-api-595cc648c-76fvf\" (UID: \"94d86bba-44b6-4065-8006-575e52434733\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:49.015074 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:49.015007 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjq8\" (UniqueName: \"kubernetes.io/projected/94d86bba-44b6-4065-8006-575e52434733-kube-api-access-kzjq8\") pod \"maas-api-595cc648c-76fvf\" (UID: \"94d86bba-44b6-4065-8006-575e52434733\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:49.064381 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:49.064349 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:49.439865 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:49.439843 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:52:49.441247 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:49.441143 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-76fvf"] Nov 03 19:52:50.022406 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:50.022370 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" event={"ID":"94d86bba-44b6-4065-8006-575e52434733","Type":"ContainerStarted","Data":"c89e07e56a7f9992300a96c965d53d4f454ae155275621f57834c7c9d262f3e7"} Nov 03 19:52:51.029378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:51.029342 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" event={"ID":"94d86bba-44b6-4065-8006-575e52434733","Type":"ContainerStarted","Data":"290c8b2ae5a4e103c51dc28f9b1db828c674fd8ffeaac00b56d038d63448fe3c"} Nov 03 19:52:51.029901 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:51.029454 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:52:51.150632 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:51.150359 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" podStartSLOduration=2.169196324 podStartE2EDuration="3.15031372s" totalImagesPullingTime="981.117396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:51.149374625 +0000 UTC m=+4105.121935804" watchObservedRunningTime="2026-11-03 19:52:51.15031372 +0000 UTC m=+4105.122874905" Nov 03 19:52:57.039795 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:52:57.039765 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-76fvf" Nov 03 19:53:05.400443 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:05.400408 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hpds8"] Nov 03 19:53:05.400933 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:05.400601 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" podUID="552330ce-c90c-42a5-b783-effafa6afb8f" containerName="maas-api" containerID="cri-o://1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48" gracePeriod=30 Nov 03 19:53:06.758926 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:06.758864 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 19:53:06.901999 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:06.901968 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552330ce-c90c-42a5-b783-effafa6afb8f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552330ce-c90c-42a5-b783-effafa6afb8f-openshift-service-ca\") pod \"552330ce-c90c-42a5-b783-effafa6afb8f\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " Nov 03 19:53:06.902171 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:06.902064 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552330ce-c90c-42a5-b783-effafa6afb8f-kube-api-access-gjfgb\" (UniqueName: \"kubernetes.io/projected/552330ce-c90c-42a5-b783-effafa6afb8f-kube-api-access-gjfgb\") pod \"552330ce-c90c-42a5-b783-effafa6afb8f\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " Nov 03 19:53:06.902171 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:06.902104 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-metrics-certs\") pod \"552330ce-c90c-42a5-b783-effafa6afb8f\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " Nov 03 19:53:06.902171 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:06.902160 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-maas-api-tls\") pod \"552330ce-c90c-42a5-b783-effafa6afb8f\" (UID: \"552330ce-c90c-42a5-b783-effafa6afb8f\") " Nov 03 19:53:06.902465 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:06.902435 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552330ce-c90c-42a5-b783-effafa6afb8f-openshift-service-ca" pod "552330ce-c90c-42a5-b783-effafa6afb8f" (UID: "552330ce-c90c-42a5-b783-effafa6afb8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:06.902647 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:06.902619 5019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552330ce-c90c-42a5-b783-effafa6afb8f-openshift-service-ca\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:06.904709 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:06.904685 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-metrics-certs" pod "552330ce-c90c-42a5-b783-effafa6afb8f" (UID: "552330ce-c90c-42a5-b783-effafa6afb8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:53:06.905069 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:06.904825 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-maas-api-tls" pod "552330ce-c90c-42a5-b783-effafa6afb8f" (UID: "552330ce-c90c-42a5-b783-effafa6afb8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:53:06.905069 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:06.904932 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552330ce-c90c-42a5-b783-effafa6afb8f-kube-api-access-gjfgb" pod "552330ce-c90c-42a5-b783-effafa6afb8f" (UID: "552330ce-c90c-42a5-b783-effafa6afb8f"). InnerVolumeSpecName "kube-api-access-gjfgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:07.003802 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:07.003751 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-maas-api-tls\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:07.003802 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:07.003795 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjfgb\" (UniqueName: \"kubernetes.io/projected/552330ce-c90c-42a5-b783-effafa6afb8f-kube-api-access-gjfgb\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:07.003802 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:07.003808 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552330ce-c90c-42a5-b783-effafa6afb8f-metrics-certs\") on node \"ip-10-0-139-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:07.156600 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:07.156507 5019 generic.go:356] "Generic (PLEG): container finished" podID="552330ce-c90c-42a5-b783-effafa6afb8f" containerID="1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48" exitCode=0 Nov 03 19:53:07.156600 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:07.156552 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" Nov 03 19:53:07.156831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:07.156586 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" event={"ID":"552330ce-c90c-42a5-b783-effafa6afb8f","Type":"ContainerDied","Data":"1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48"} Nov 03 19:53:07.156831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:07.156650 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hpds8" event={"ID":"552330ce-c90c-42a5-b783-effafa6afb8f","Type":"ContainerDied","Data":"4a88ebfc3d5c3a164b979fe65e095547fae121b0072d4b5ea90f9611ee48ae74"} Nov 03 19:53:07.156831 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:07.156681 5019 scope.go:122] "RemoveContainer" containerID="1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48" Nov 03 19:53:07.167209 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:07.167189 5019 scope.go:122] "RemoveContainer" containerID="1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48" Nov 03 19:53:07.167538 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:53:07.167519 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48\": container with ID starting with 1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48 not found: ID does not exist" containerID="1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48" Nov 03 19:53:07.167597 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:07.167549 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48"} err="failed to get container status \"1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48\": rpc error: code = NotFound desc = could not find container \"1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48\": container with ID starting with 1c3c00444c76b41f99bc080749234a53dc312e46b71baf48449a8f6ef9431f48 not found: ID does not exist" Nov 03 19:53:07.324872 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:07.324839 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hpds8"] Nov 03 19:53:07.376490 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:07.376454 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hpds8"] Nov 03 19:53:08.149232 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:53:08.149191 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552330ce-c90c-42a5-b783-effafa6afb8f" path="/var/lib/kubelet/pods/552330ce-c90c-42a5-b783-effafa6afb8f/volumes" Nov 03 19:53:11.310347 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:53:11.310171 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:53:41.296222 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:53:41.296180 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:54:11.296888 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:54:11.296811 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:54:14.973798 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:14.973767 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d4c5c4549-hsf4r_8cdf44f6-ad95-4a5c-b11f-73c70208f9ec/manager/0.log" Nov 03 19:54:16.350022 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:16.349993 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bbcd964c6-ztfsd_007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef/manager/0.log" Nov 03 19:54:17.989025 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:17.988992 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff9d8584c-pkhrx_21aa73e8-9d1f-4b5f-a130-6bbab3ee085e/manager/0.log" Nov 03 19:54:18.073532 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:18.073502 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff9d8584c-pkhrx_21aa73e8-9d1f-4b5f-a130-6bbab3ee085e/copy-manifests/0.log" Nov 03 19:54:18.305733 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:18.305653 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-msvrv_e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f/proxy/0.log" Nov 03 19:54:19.111841 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:19.111810 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-76fvf_94d86bba-44b6-4065-8006-575e52434733/maas-api/0.log" Nov 03 19:54:21.083008 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:21.082981 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-r6l7j_550a4d2f-33de-42bc-9a90-d89dbc7771b7/postgres/0.log" Nov 03 19:54:23.308078 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:23.308046 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-kcfwm_db7c9665-d048-4b53-8412-d91f81854523/manager/0.log" Nov 03 19:54:23.790377 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:23.790316 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-p5zkj_11b09062-eabd-4950-875a-a0fd97ab66ee/manager/0.log" Nov 03 19:54:24.331122 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:24.331083 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-2mzjt_735187df-3794-4b77-a6f2-9ee7ea5803b4/manager/0.log" Nov 03 19:54:25.363378 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:25.363349 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-fwkh2_9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c/discovery/0.log" Nov 03 19:54:25.587709 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:25.587674 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dc56b666-v5sf5_58353d70-ebc7-4ecc-b429-ff9acdecb687/kube-auth-proxy/0.log" Nov 03 19:54:26.026705 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:26.026675 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-4hpg8_7ab5eebd-1d3e-4390-b501-0598458c550e/istio-proxy/0.log" Nov 03 19:54:26.259836 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:26.259813 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-75zmw_eb93eda2-29b3-439e-a19d-984ecad5177f/istio-proxy/0.log" Nov 03 19:54:26.479436 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:26.479409 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-bkdvr_0a8164ba-f2ef-4586-9ee6-2d6c981c1f36/istio-proxy/0.log" Nov 03 19:54:26.713905 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:26.713878 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-c699g_b3355b2e-2808-446c-be03-26729210bfb6/istio-proxy/0.log" Nov 03 19:54:28.273267 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:28.273239 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-txjmz_db160175-01ea-4325-b1e7-60249341c899/payload-pre-processing/0.log" Nov 03 19:54:28.707974 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:28.707937 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8dd798ffc-wswht_d8c875c2-ea97-435e-821a-067bc0d5c58e/router/0.log" Nov 03 19:54:29.741799 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:29.741773 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-lzknn_64142b5e-3faa-403f-a7b7-60638cfeb7fc/main/0.log" Nov 03 19:54:29.967204 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:29.967167 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-vspbc_fc9e70c5-7751-4377-aacf-3c8c8530bdc9/main/0.log" Nov 03 19:54:29.987099 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:29.987072 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:54:30.001722 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:30.001661 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-48wh4_d40ca2ad-bedb-4aa4-b1fd-3ad6da24fcc1/console-operator/2.log" Nov 03 19:54:30.434280 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:30.434250 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp_2704fa8b-f7ed-49ec-918c-45d4df1cc1dc/main/0.log" Nov 03 19:54:40.846026 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:40.845998 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d4c5c4549-hsf4r_8cdf44f6-ad95-4a5c-b11f-73c70208f9ec/manager/0.log" Nov 03 19:54:41.295546 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:54:41.295512 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:54:41.987900 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:41.987871 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bbcd964c6-ztfsd_007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef/manager/0.log" Nov 03 19:54:43.591363 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:43.591315 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff9d8584c-pkhrx_21aa73e8-9d1f-4b5f-a130-6bbab3ee085e/copy-manifests/0.log" Nov 03 19:54:43.656810 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:43.656780 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff9d8584c-pkhrx_21aa73e8-9d1f-4b5f-a130-6bbab3ee085e/manager/0.log" Nov 03 19:54:43.870298 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:43.870231 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-msvrv_e2be4b9a-ec7c-4c2c-a3d2-4cce86c9a74f/proxy/0.log" Nov 03 19:54:44.427773 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:44.427740 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-76fvf_94d86bba-44b6-4065-8006-575e52434733/maas-api/0.log" Nov 03 19:54:46.239561 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:46.239534 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-r6l7j_550a4d2f-33de-42bc-9a90-d89dbc7771b7/postgres/0.log" Nov 03 19:54:47.962834 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:47.962792 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-kcfwm_db7c9665-d048-4b53-8412-d91f81854523/manager/0.log" Nov 03 19:54:48.433214 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:48.433183 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-p5zkj_11b09062-eabd-4950-875a-a0fd97ab66ee/manager/0.log" Nov 03 19:54:48.945942 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:48.945915 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-2mzjt_735187df-3794-4b77-a6f2-9ee7ea5803b4/manager/0.log" Nov 03 19:54:49.778844 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:49.778816 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-fwkh2_9f8f46e0-cb3c-4ef7-a1bd-673150b5e80c/discovery/0.log" Nov 03 19:54:50.009925 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:50.009896 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dc56b666-v5sf5_58353d70-ebc7-4ecc-b429-ff9acdecb687/kube-auth-proxy/0.log" Nov 03 19:54:50.449549 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:50.449520 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-4hpg8_7ab5eebd-1d3e-4390-b501-0598458c550e/istio-proxy/0.log" Nov 03 19:54:50.675810 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:50.675783 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-75zmw_eb93eda2-29b3-439e-a19d-984ecad5177f/istio-proxy/0.log" Nov 03 19:54:50.901928 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:50.901852 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-bkdvr_0a8164ba-f2ef-4586-9ee6-2d6c981c1f36/istio-proxy/0.log" Nov 03 19:54:51.125906 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:51.125872 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-c699g_b3355b2e-2808-446c-be03-26729210bfb6/istio-proxy/0.log" Nov 03 19:54:52.757184 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:52.757156 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-txjmz_db160175-01ea-4325-b1e7-60249341c899/payload-pre-processing/0.log" Nov 03 19:54:53.202595 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:53.202569 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8dd798ffc-wswht_d8c875c2-ea97-435e-821a-067bc0d5c58e/router/0.log" Nov 03 19:54:54.054523 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:54.054490 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-lzknn_64142b5e-3faa-403f-a7b7-60638cfeb7fc/main/0.log" Nov 03 19:54:54.296253 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:54.296223 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-vspbc_fc9e70c5-7751-4377-aacf-3c8c8530bdc9/main/0.log" Nov 03 19:54:54.747198 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:54:54.747167 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kndlp_2704fa8b-f7ed-49ec-918c-45d4df1cc1dc/main/0.log" Nov 03 19:55:11.297458 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:55:11.297425 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:55:26.962152 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:55:26.962120 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bbcd964c6-ztfsd_007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef/manager/0.log" Nov 03 19:55:27.548741 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:55:27.548709 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bbcd964c6-ztfsd_007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef/manager/0.log" Nov 03 19:55:41.295830 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:55:41.295741 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:56:07.114549 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:07.114514 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bbcd964c6-ztfsd_007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef/manager/0.log" Nov 03 19:56:07.695862 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:07.695830 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bbcd964c6-ztfsd_007e5bc7-ebd6-49e0-94d1-e0a22d70d7ef/manager/0.log" Nov 03 19:56:11.295565 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:56:11.295532 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:56:18.052358 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:18.052253 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fn8l9_dcd42898-f7cc-4198-851b-db51f7ba110a/global-pull-secret-syncer/0.log" Nov 03 19:56:19.220446 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:19.220415 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-gjwzc_48738200-5d6d-4a3e-8798-700496cd1681/connection-checker/0.log" Nov 03 19:56:19.630220 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:19.630194 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ht2s6_a6fe8230-18dc-48a4-aa96-d742e707222d/connection-checker/0.log" Nov 03 19:56:20.045063 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:20.045036 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-l6xkx_eedc3daa-f8aa-4c65-a525-d5072215f3e7/connection-checker/0.log" Nov 03 19:56:20.909754 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:20.909728 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dwp5g_9126b699-4c1b-46ca-976c-a4e1813357a1/konnectivity-agent/0.log" Nov 03 19:56:22.531619 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:22.531587 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-179.us-west-2.compute.internal_b1997d138e76444593b76299593de1e7/haproxy/0.log" Nov 03 19:56:27.970410 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:27.970365 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-kcfwm_db7c9665-d048-4b53-8412-d91f81854523/manager/0.log" Nov 03 19:56:28.012533 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.012502 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t"] Nov 03 19:56:28.012946 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.012934 5019 cpu_manager.go:414] "containerMap: removing container" podUID="552330ce-c90c-42a5-b783-effafa6afb8f" containerName="maas-api" Nov 03 19:56:28.012989 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.012946 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552330ce-c90c-42a5-b783-effafa6afb8f" containerName="maas-api" Nov 03 19:56:28.013054 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.013044 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="552330ce-c90c-42a5-b783-effafa6afb8f" containerName="maas-api" Nov 03 19:56:28.016195 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.016180 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.050930 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.050902 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef5c9688-523a-4a1b-acac-fe41e9386cf1-proc\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.051050 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.050939 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrs5\" (UniqueName: \"kubernetes.io/projected/ef5c9688-523a-4a1b-acac-fe41e9386cf1-kube-api-access-bwrs5\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.051095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.051080 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef5c9688-523a-4a1b-acac-fe41e9386cf1-podres\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.051147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.051106 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef5c9688-523a-4a1b-acac-fe41e9386cf1-lib-modules\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.051147 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.051129 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef5c9688-523a-4a1b-acac-fe41e9386cf1-sys\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.069804 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.069777 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t5k7x\"/\"kube-root-ca.crt\"" Nov 03 19:56:28.121364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.121316 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t5k7x\"/\"openshift-service-ca.crt\"" Nov 03 19:56:28.121364 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.121316 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t5k7x\"/\"default-dockercfg-c8qsb\"" Nov 03 19:56:28.123615 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.123592 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t"] Nov 03 19:56:28.152032 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.152003 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef5c9688-523a-4a1b-acac-fe41e9386cf1-podres\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.152181 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.152125 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef5c9688-523a-4a1b-acac-fe41e9386cf1-lib-modules\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.152181 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.152167 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef5c9688-523a-4a1b-acac-fe41e9386cf1-sys\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.152305 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.152133 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef5c9688-523a-4a1b-acac-fe41e9386cf1-podres\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.152305 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.152246 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef5c9688-523a-4a1b-acac-fe41e9386cf1-lib-modules\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.152305 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.152250 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef5c9688-523a-4a1b-acac-fe41e9386cf1-proc\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.152305 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.152299 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef5c9688-523a-4a1b-acac-fe41e9386cf1-proc\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.152520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.152304 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrs5\" (UniqueName: \"kubernetes.io/projected/ef5c9688-523a-4a1b-acac-fe41e9386cf1-kube-api-access-bwrs5\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.152520 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.152310 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef5c9688-523a-4a1b-acac-fe41e9386cf1-sys\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.214406 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.214381 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrs5\" (UniqueName: \"kubernetes.io/projected/ef5c9688-523a-4a1b-acac-fe41e9386cf1-kube-api-access-bwrs5\") pod \"perf-node-gather-daemonset-rxr4t\" (UID: \"ef5c9688-523a-4a1b-acac-fe41e9386cf1\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.328249 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.328173 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:28.699393 ip-10-0-139-179 kubenswrapper[5019]: W1103 19:56:28.699360 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podef5c9688_523a_4a1b_acac_fe41e9386cf1.slice/crio-a5b14a52acedb04c6528e8b4aeb0786496d1a2bcb59ff00a4ffcb0df30033606 WatchSource:0}: Error finding container a5b14a52acedb04c6528e8b4aeb0786496d1a2bcb59ff00a4ffcb0df30033606: Status 404 returned error can't find the container with id a5b14a52acedb04c6528e8b4aeb0786496d1a2bcb59ff00a4ffcb0df30033606 Nov 03 19:56:28.702655 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.702619 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t"] Nov 03 19:56:28.802730 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:28.802705 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-p5zkj_11b09062-eabd-4950-875a-a0fd97ab66ee/manager/0.log" Nov 03 19:56:29.303624 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:29.303588 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" event={"ID":"ef5c9688-523a-4a1b-acac-fe41e9386cf1","Type":"ContainerStarted","Data":"af1cae49699b5e702328329e1a6b7388114b974fc65778956222e9753f4f078b"} Nov 03 19:56:29.304018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:29.303644 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" event={"ID":"ef5c9688-523a-4a1b-acac-fe41e9386cf1","Type":"ContainerStarted","Data":"a5b14a52acedb04c6528e8b4aeb0786496d1a2bcb59ff00a4ffcb0df30033606"} Nov 03 19:56:29.304018 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:29.303672 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:29.422013 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:29.421964 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" podStartSLOduration=2.421950931 podStartE2EDuration="2.421950931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:29.421758365 +0000 UTC m=+4323.394319546" watchObservedRunningTime="2026-11-03 19:56:29.421950931 +0000 UTC m=+4323.394512112" Nov 03 19:56:29.627812 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:29.627732 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-2mzjt_735187df-3794-4b77-a6f2-9ee7ea5803b4/manager/0.log" Nov 03 19:56:34.987133 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:34.987093 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-4dlvx_f0725d29-b4bf-4d33-8089-2c568ba41b0f/cluster-monitoring-operator/0.log" Nov 03 19:56:35.318095 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:35.318021 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-rxr4t" Nov 03 19:56:38.625987 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:38.625926 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b6r9c_f924bab6-aae8-479e-aa28-72bac7cf6b2f/node-exporter/0.log" Nov 03 19:56:39.029258 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:39.029236 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b6r9c_f924bab6-aae8-479e-aa28-72bac7cf6b2f/kube-rbac-proxy/0.log" Nov 03 19:56:39.431255 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:39.431230 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b6r9c_f924bab6-aae8-479e-aa28-72bac7cf6b2f/init-textfile/0.log" Nov 03 19:56:41.296281 ip-10-0-139-179 kubenswrapper[5019]: E1103 19:56:41.296247 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w77sj" containerName="extractor" Nov 03 19:56:45.157281 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:45.157248 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-tm5fw_b5ebef9a-b419-487e-92b3-ed9344044974/prometheus-operator/0.log" Nov 03 19:56:45.554533 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:45.554487 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-tm5fw_b5ebef9a-b419-487e-92b3-ed9344044974/kube-rbac-proxy/0.log" Nov 03 19:56:45.957814 ip-10-0-139-179 kubenswrapper[5019]: I1103 19:56:45.957781 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-9njx8_357d8534-0165-47dc-98ba-9e56da679555/prometheus-operator-admission-webhook/0.log"